Researchers say these AI models tend to skip proper clinical reasoning and jump too quickly to conclusions.
Marc Succi from Massachusetts General Hospital put it simply: "Despite continued improvements, off-the-shelf large language models are not ready for unsupervised clinical-grade deployment,"
Still, about 66 million Americans are using AI for health advice, with some citing cost, time, or access barriers.
But here's the catch: one in 10 users have gotten potentially unsafe advice from these tools, highlighting why good oversight is so important as more people turn to AI for health questions.