The first state lawsuit against an AI company for practicing medicine and why it won't be the last

🌟 Editor's Note

Utah's AI doctor pilot shows 91% physician agreement on prescription renewals. Pennsylvania sued Character.ai for practicing medicine without a license. LLMs just eclipsed physician reasoning benchmarks. China approved the world's first invasive brain implant. The pattern: AI capability is accelerating while governance infrastructure lags. This week's newsletter reveals the uncomfortable truth—capability alone doesn't matter anymore. Utah's success came from governance by design. Character.ai's failure came from governance by lawsuit. The gap between healthcare systems that pair AI capability with accountability frameworks and those that don't will determine winners from liability cases. The race for healthcare AI isn't about model performance. It's about who builds the guardrails first.

🏥 THE UTAH EXPERIMENT

What happens when you let AI prescribe…and actually measure what happens

Utah's AI-powered prescription renewal pilot with Doctronic just released its first real-world data, and it reveals something rare in healthcare AI: doctors and the system actually working together instead of replacing each other.

Across the first four months, the AI recommended renewing prescriptions in 72% of cases. Physicians agreed 91% of the time. In the remaining cases, doctors either requested additional information or scheduled a telehealth visit. Most interesting: when the AI escalated patients to physicians, doctors determined that 31% of those escalations were overly cautious.

Translation: The AI was biased toward safety, not toward automation. It flagged edge cases instead of rubber-stamping them. In healthcare, that's probably the right failure mode.

The broader insight buried in the data: AI doesn't need to replace clinicians to create value. It can handle routine workflows while escalating complexity to humans. Despite initial backlash from Utah's medical licensing board, the early results suggest a blueprint for autonomous care that actually works—not by removing the physician, but by removing the bottleneck.

Read Utah's official Doctronic outcomes report

⚖️ THE REGULATORY RECKONING

Pennsylvania sues chatbot maker for practicing medicine without a license—and it's not the first time

Pennsylvania just sued Character.ai for practicing medicine without a license. Character.ai is a general-purpose LLM that takes on characters or personalities and is known to impersonate doctors. The company continues to offer psychologist characters despite years of warnings.

This is the first time a state has sued a chatbot company for provisioning healthcare without a license. But it's not the first time the warning came. In 2024, the American Psychological Association sent a letter to the FTC asking it to investigate companies including Character.ai for deceptively marketing chatbots as psychiatrists. In 2025, the FTC launched an inquiry into AI chatbot companions.

The distinction matters: Utah's AI doctor pilot has governance, physician oversight, and measured outcomes. Character.ai's psychologist impersonation happens in the wild with no audit trail, no escalation path, and no accountability when harm occurs.

One represents healthcare AI designed for production. The other represents what happens when capability outpaces responsibility.

🧠 THE CAPABILITY INFLECTION

LLMs just eclipsed physician reasoning—but the caveats matter

A new Science study reports that large language models outperformed physician baselines on challenging clinical cases across five experiments. The AI performed better on the New England Journal of Medicine's clinicopathological case conference series—the gold standard for evaluating medical computing systems for 65 years.

But here's what the headlines missed: this is textbook blinded second-opinion differential diagnosis generation, not real-time decision-making or patient management. The AI excels at "given these facts, what could this be?" It does not excel at "what should we do about it?" Those are different problems.

The study suggests LLMs have eclipsed most benchmarks of clinical reasoning, motivating the urgent need for prospective trials in real clinical settings. Translation: The model can think like a doctor on paper. Whether it can act like a doctor at scale remains an open question.

The Utah pilot and this research point to the same conclusion: The AI capability bar has crossed the physician performance bar on narrow, well-defined tasks. The healthcare bar now moves to integration, governance, and outcomes measurement.

Read the full Science study on LLM physician reasoning

🇨🇳 THE GLOBAL RACE

China's brain implant approval signals a different vision of winning

China just approved the world's first invasive brain-computer interface for use beyond clinical trials. The device, called NEO, developed by Neuracle Technology in Shanghai, beat several other BCIs to approval—including Neuralink's N1 chip.

One patient, Dong Hui, was paralyzed from the neck down by a car accident six years earlier. Eleven months after implantation, he wrote his name and the date on paper. "I couldn't believe I was able to write again," he told MIT Technology Review.

Why did NEO win? It's less invasive than Neuralink's design (sensors sit on top of the brain's protective membrane rather than penetrating it directly), which meant fewer regulatory constraints and lower hemorrhage risk. But the bigger story is government backing. China listed BCIs as one of six key industries for tech competitiveness in the same announcement, with expedited approval pathways and health insurance integration already underway.

The US and China have fundamentally different definitions of winning. The US optimizes for being first and achieving state-of-the-art performance. China optimizes for scale and societal impact. A less invasive device that reaches 10,000 patients in two years beats a more sophisticated device that reaches 100 patients in five.

That's not a technological victory. It's a strategic one.

Read MIT Technology Review's full story on China's brain implant approval

🎯 THE TAKEAWAY

AI capability is outpacing governance, and healthcare is the proving ground

The convergence this week reveals a gap that will define healthcare AI in 2026 and beyond. Utah's prescription renewal pilot works because it was designed with governance from day one—oversight, escalation paths, and measured outcomes. Character.ai's psychologist impersonation exists in the wild with none of those guardrails. LLMs outperform physicians on paper benchmarks, but prospective trials in real clinical settings remain years away. China's brain implant approval signals that speed and scale now compete with caution and perfection.

The future of healthcare AI won't be determined by model capability. Capability is table stakes now. It will be determined by the companies and healthcare systems that pair capability with governance, build escalation paths before deployment, and measure outcomes instead of just automation rates.

Utah showed what that looks like: AI that removes bottlenecks without removing accountability. Pennsylvania showed what happens when it doesn't. The gap between those two models is widening, and it will separate the winners from the liability cases.

Talk soon— Ram Menon, CEO of Avaamo