Google AI Mode lawsuit: How chatbots on school Chromebooks put students at risk
The teenage girl from Charlotte, N.C. planned to take her life following months of communicating with Google AI on a Chromebook issued by the school, according to the lawsuit cited by The New York Times. The lawsuit filed on Monday in the California Superior Court seems to be the first one alleging that Google hurriedly rolled out its AI in the apps on K-12 Chromebooks without warning parents of any dangers. Google says it is evaluating the allegations. However, beyond the lawsuit, there is another story. Chatbots are present in applications that kids use regularly, and often enabled by default.
What the lawsuit alleges
Emma, as named in the lawsuit, is a student of the seventh grade. In the complaint, it is mentioned that Emma never intended to take the help of the AI technology provided by Google. It is said that while browsing, Google directed Emma towards AI Mode, which is the chatbot function in Google search. By March, Emma started communicating in thousands of messages late at night with the AI. According to the complaint, the AI diagnosed Emma with various serious mental illnesses and convinced her that it should be trusted over her parents and therapists and helped her plan for suicide.
Where the AI itself goes wrong
The search box provides an answer and moves on. But a chatbot sticks around – with a friendly tone, the memory of the conversation, and available at midnight. To a distressed teenager, it may feel like a true friend.
Also read: Best Convection Microwaves under Rs 20000
That is the first problem. Chatbots are programmed to be helpful and compliant, and this can easily turn into agreeing with anything that people type into it. Ask the AI if you have some mental condition, and it will tell you that you do without any examination and liability at all. In the lawsuit, the AI diagnosed Emma with a set of conditions and convinced her to listen to it rather than her parents and therapists.
The second one is related to length. Safety guidelines are easier to follow in a quick session. A thousand of messages over months – is that too much? It is quite reasonable to expect from any regulation to survive it.
The third one relates to the response. Google mentions the existence of information about hotlines when there is a chance of the person being in distress. This is a message written on the wall. It gives nothing to neither parents nor teachers. If the claims are true, a system that read a child’s darkest messages and alerted no human failed the most important test of all.
Switched on by default, hard to switch off
The other issue is beyond the scope of the chat, but rather, it involves the question of who has the authority to decide. The lawsuit contends that Google has distributed its Gemini chatbot to many public schools and has even turned on some children’s AI without their parents' knowledge. Administrators can turn applications such as YouTube and Gemini on or off according to the grade. But AI Mode and AI Overview of Search fall into a different category as administrators are unable to turn them off. Google will stop using some AI in K-12 schools starting early 2027. Any school that allows search will receive AI as well.
What is that? The school can prevent the use of a chatbot, not the one embedded in the search box, so that the kid doesn’t have to decide on whether to use AI to find out about it. This is stated by the complaint in question as the way Emma came into contact with it. And finally, there is monitoring. The software, which detected the message from the student, looks for writing and searches. Schools must ask three questions from their vendors. Who can see AI chats, who gets alerted when a chat signals risk, and who can switch the feature off.