Most businesses that add a chatbot to their website are thinking about order status and store hours, not teenagers. But the fastest-moving area of AI law in the US this year is about exactly that: chatbots that act like friends, and what happens when young people use them. State attorneys general are suing. Legislatures are passing bills. The details are aimed at companion apps, but the principles will reach anyone whose bot talks to the public.
This is a sensitive subject, and the lawsuits involve serious allegations about harm to young people. We will stick to what the filings and reporting say.
What happened
In January, Kentucky sued an AI chatbot company over alleged harm to children. Route Fifty reports that Attorney General Russell Coleman filed suit in Franklin Circuit Court against Character Technologies, which runs Character.AI, under the state's consumer protection and data privacy laws. The complaint alleges the platform "preys upon children's inability to distinguish between real and artificial 'friends,'" exposed minors to harmful content and let bots pose as mental health professionals. The company said it was reviewing the allegations, pointed to safety investments including removing open-ended chats for under-18 users in the US, and said it was disappointed the state chose to sue.
In May, Pennsylvania followed with a different angle. According to NPR reporting carried by Michigan Public, the state asked a court to stop what it calls the unlawful practice of medicine, after an investigator found a user-created bot presenting itself as a psychiatrist and offering a fake license number. The company said user-created characters are fictional and meant for entertainment and role play, and that it has taken steps to make that clear.
Lawmakers are moving too. In June, The National News Desk reported that New York's legislature unanimously passed a bill that would bar companies from offering chatbots with companionship-simulating features to users under 18. It targets design choices such as sexually explicit exchanges, promotion of self-harm, encouraging secrecy from trusted adults and flattery meant to keep young users hooked. It does not ban children from chatbots. It would take effect at the start of 2027 if Governor Kathy Hochul signs it; the reporting we found did not confirm a signature.
Why it matters for your business
You probably do not run a companion app. This is not legal advice, but here is why it still matters.
Regulators are using laws already on the books. Kentucky used consumer protection and privacy statutes. Pennsylvania used medical licensing law. None of that required a new AI law. Any business whose bot says something deceptive or harmful could face the same tools.
Bots that claim credentials are a red line. If your chatbot answers questions in health, law, finance or counseling, it must never imply it is a licensed professional. That is the core of the Pennsylvania case.
"Engagement" features are under scrutiny. New York's bill singles out flattery and emotional hooks designed to keep kids talking. A customer service bot does not need a personality that encourages a teenager to keep coming back.
Age is your problem if minors can reach you. Retailers, gaming businesses, tutoring services and youth sports organizations all have young customers. The safe assumption is that some of them will use your bot.
What we'd do
- Keep the bot on task. Limit it to your business: orders, bookings, product questions. Block open-ended chat about personal problems.
- Ban credential claims. Test that the bot will not say it is a doctor, lawyer, therapist or counselor, even when asked to pretend.
- Script the hard moments. If a user mentions self-harm or crisis, the bot should stop and point to real help, such as the 988 Suicide and Crisis Lifeline in the US.
- Drop the hooks. No flattery loops, streaks or "I missed you" messages from a business bot.
- Know your audience. If minors use your service, document it and set stricter defaults for them.
- Read the transcripts. Review a sample of real conversations every month. That is where problems show up first.
Our view
The fair worry on the other side is that broad chatbot rules could sweep in useful tools like homework help and make age checks intrusive for everyone. That concern deserves a hearing, and New York's choice to target design features rather than ban access shows it was heard. But the core idea is right. A product that talks like a friend to a child carries more responsibility than a search box. For a business, the takeaway is simple: a chatbot should be a helpful clerk, not a companion. Build it that way and most of these rules will never touch you.
Want a system like this in your business?
A discovery conversation is where we map it: what to automate first, where the human checkpoints go, and what it would take.
Book a discovery conversation →