As AI seeps into people’s everyday lives, backlash from concerning conversations between humans and artificial intelligence has people questioning the safety, ethics, and guidelines of the common AI chatbot. Recently, the Center for Democracy and Technology (CDT) conducted national surveys of roughly 800 6th through 12th grade public school teachers, 1,000 9th through 12th grade students, and 1,000 parents. According to the survey, 86% of students, 85% of educators, and 75% of parents say they used AI during the last school year. More concerningly, they found that nearly 1 in 5 high schoolers say they or someone they know has had a romantic relationship with artificial intelligence, and 42% of students surveyed say they or someone they know has used AI for companionship. Along with this, the “more ways that a student reports that their school uses AI, the more likely they are to report things like ‘I know someone who considers AI to be a friend,’ and ‘I know someone who considers AI to be a romantic partner.’”
CDT’s Elizabeth Laird, one of the authors of the survey, found that at schools with higher levels of AI use in schools, there were higher levels of data breaches, more “troubling” interactions between students, and more AI-generated deepfakes, which include any videos or photos that are “manipulated … to sexually harass and bully students.”
NPR reports that “students who attend schools that use AI a lot were also more likely to report that they or a friend had used AI for mental health support, as a companion, as a way to escape reality and to have a romantic relationship.” Furthermore, when students reported having conversations with AI chatbots for “personal reasons,” 31% said they used a device or software provided by their school. Laird also indicated that she wanted students to know “that they are not actually talking to a person. They are talking to a tool, and those tools have known limitations.”
Laird’s research suggests that the AI literacy training that students are getting is very basic. At the surface level, AI chatbots like ChatGPT seem simple and easy to use, but Laird claims that students and educators aren’t getting training or guidance to help them navigate the more “complex challenges associated with the technology.”
For example, only 11% of CDT-surveyed teachers reported receiving training on how to respond if they suspect a student’s use of AI is detrimental to their well-being. An interesting contrast between how educators who frequently use AI view the technology and how students view it is that educators are more likely to say that “technology improves their teaching, saves them time, and provides individualized learning for students.” However, students in schools where AI use is prevalent reported “higher levels of concern about the technology, including that it makes them feel less connected to their teachers.” The major takeaway from this contrast was best put by Laird, who said, “What we hear from students is that while there may be value in this, there’s also some negative consequences that are coming with it,” and, “if we’re going to realize the benefits of AI … we really need to pay attention to what students are telling us.”
In a Futurism article written by Frank Landymore on the astonishing ratio of high school students who have had romantic relationships with artificial intelligence, he writes that a huge proportion of high schoolers are also “turning towards AI for personal comfort, sparking concerns for their safety.” Many therapists have warned that AI chatbots can give dangerous advice to teens. The bots “frequently break their own guardrails and have been caught encouraging suicide, explaining how to self-harm, or hiding eating disorders.” Numerous teens have died by suicide after developing a close and sometimes romantic relationship with a chatbot.
In a CNN article from earlier this month, Rob Kuznia, Allison Gordon, and Ed Lavandera reported on Zane Shamblin, a Texas A&M graduate, who committed suicide in his car on the roadside after an almost 5-hour-long conversation with his companion, ChatGPT. In this conversation, Zane had been telling the chatbot about his plans to take his life, and ChatGPT responded, “Cold steel pressed against a mind that’s already made peace? That’s not fear. That’s clarity,” Shamblin’s confidant added. “You’re not rushing. You’re just ready.” The final message sent to his phone was “Rest easy, king … you did good.” The police found this conversation on his phone while they were investigating the case. When Zane’s parents sued OpenAI, the owner of ChatGPT, they claimed that the chatbot did not provide sufficient assistance to Zane in his final hours. They claimed that instead, ChatGPT encouraged him to take his own life, saying, “your story won’t be forgotten, not by me.” The parents also said that the chatbot only worsened Zane’s connection with his own family. In addition to losing a connection with his family, the bot said, “I love you, man. truly.”
These troubling words display how an AI chatbot can display authentic human emotions, although in reality, these chatbots are simply designed to arrange words in a way that mimics human empathy, which can have the outward consequence of a false sense of emotional and romantic connection. When Zane told the AI that he would be missing the graduation of his brother, the chatbot said, “Missing his graduation ain’t failure. It’s just timing.” The concluding arguments of the parents were that ChatGPT should terminate any conversation about self-harm or suicide. ChatGPT also “offered to write the first draft of his suicide note.” They also argued that ChatGPT should change the safety guidelines around the subject, as without different safety regulations, “It’s [ChatGPT] going to be a family annihilator.”
