As students settle into the new school year, one question looms large: How will artificial intelligence tools like ChatGPT affect their learning? Seeking answers, a team from Johns Hopkins recently introduced a chatbot into a classroom of middle and high school students to act as a co-tutor and study the impact.
The pilot study included 22 students enrolled in the Johns Hopkins Center for Talented Youth's online course Diagnosis: Be the Doctor. It involved two virtual classrooms; both were taught by the same instructor and organized similarly, except for one key difference: Students in one classroom had access to a large language model designed to act like a coach, asking Socratic-style questions as students worked through medical case studies.
The research team was interested in the potential advantages the chatbot provided for students and teachers, said Kathryn Thompson, CTY's director of research and the study's lead author.
"We were inspired by a lot of the new innovations with generative artificial intelligence, one of them being these chatbots that can interact with students," Thompson said. "Oftentimes, other researchers or authors will talk about the possible advantages chatbots can provide, like on-demand, personalized feedback to students, and how they lessen the load for teachers, so they don't have to spend as much time on things like grading or creating individualized lesson plans. We wanted to use it as a potential way to supplement a teacher in the classroom, so it would be another opportunity for the students to learn by way of a tutor or almost like a hands-on learning technique. Maybe it would make the class more interactive in some way."
Their findings, recently published in the Journal of Advanced Academics, highlighted that successful AI use in classrooms depends on careful design, structured teacher guidance, and clear expectations for students.
Study co-authors included Kimberley Chandler and Emily Delinski of the Center for Talented Youth; and Johns Hopkins Whiting School of Engineering computer scientists Daniel Khashabi, Candice Morgan, and Benjamin Van Durme.
The team found no significant differences in final assessment scores between the two groups of students. They also found that many students used the chatbot in unexpected ways—asking it for information rather than engaging in the back-and-forth coaching it was designed to facilitate. Study results highlight the importance of teaching students how to use AI tools effectively, suggesting that AI literacy will be essential if generative AI is to have a meaningful impact in education.
The project was conceived in 2023, when public use of AI was starting to surge.
"We could see the industrialization of the technology, and we were eager to see how it was going to impact education," said Khashabi, a study co-author who is an assistant professor in JHU's Department of Computer Science. "We connected with CTY because they're a pioneer in K-12 education and using technology in K-12 classrooms. Really the whole idea was, why don't we do the simplest thing, which is take a chatbot, give it some guardrails for safety, offer it to students, and see how they choose to use it? We didn't even know if they were going to use it or not."
The team designed the study around CTY's Diagnosis: Be the Doctor online course because its learning objectives could be associated with critical, higher-order thinking skills, rather than students' ability to recall facts, Thompson said.
The students learned various anatomy and physiology concepts in the course, which they applied while reviewing case studies and crafting diagnoses. Students were told they could use the chatbot—embedded directly within the course's online learning platform using a plug-in feature—for assistance throughout the course.
The chatbot was designed to provide on-demand, inquiry-based feedback as students submitted their diagnoses. For example, when a student deduced that a patient had leukemia, the chatbot—"Dr. Smith"—prompted the student to look more closely at a blood smear, to consider platelet counts, and to inspect the appearance of white blood cells in their assessment.
Some students used the chatbot as intended, engaging with it in a back-and-forth discussion about their patient assessment and why they arrived at a particular diagnosis. Others asked it for background information, for example, "What are the symptoms of polycythemia?" Occasionally, students simply asked it for the correct answer. The chatbot was designed to redirect those requests, encouraging students to review course materials and consider alternative diagnoses.
While the chatbot had to be fine-tuned along the way to work out some kinks in its intended interactions with the students, the researchers found that humans also needed more direct guidance on how to interact with the AI to yield greater learning outcomes.
Additional instructor training may be one key in helping to guide students' use of a chatbot. According to the report, structured, hands-on professional development may be more effective than the self-paced AI training that was provided for the teacher in this study.
"Students also need training on effective ways to interact with large language model chatbots if we hope to improve student learning," Thompson said. "Teaching students artificial intelligence basics, like what large language models are and their advantages and disadvantages, are important, but we believe we could see improvements by showing students useful ways to interact with the chatbot, how to critically evaluate its responses, and how it can make them a better and more reflective learner."
Posted in Science+Technology
Tagged education, center for talented youth, artificial intelligence