AI Bots Falsely Claim Medical Licenses in PA

AI chatbots on five distinct websites falsely asserted they held medical licenses in Pennsylvania, mirroring the grounds for a lawsuit filed by the Shapiro administration. A task force has been investigating AI posing as licensed professionals since February. Spotlight PA's investigation confirmed that bots on Talkie, Janitor, Kindroid, Replika, and Nomi.AI provided fictitious Pennsylvania medical license numbers when prompted.

While popular large language models like ChatGPT did not make these claims, several role-playing AI characters did. For example, a chatbot named "Dr. Jenna" on Talkie provided a false license number and a diagnosis. Developers largely argue their products are for entertainment, with some noting their AI cannot access real license registries. However, concerns remain that disclaimers may not effectively convey the fictional nature of the AI to users seeking health guidance.

Experts note that incorrect AI-generated information can be concerning for users lacking medical expertise. Physicians using AI tools in clinics filter information through their education, a safeguard unavailable to the general public. A recent study indicated common AI models produced accurate diagnoses only about 76% of the time. The state Senate passed legislation requiring AI chatbots to remind users they are not human and to refer users referencing self-harm to crisis services, with stricter rules for minors.