Is AI ready to take over your prescriptions? Doctors are wary of Utah's automated refill program

But it’s also a precedent-shattering milestone that has set off alarm bells for doctors, lawyers and public health experts. The pilot program has laid bare a host of questions about the role of AI in medicine, including how it should be regulated, whether doctors should be able to veto it, and what kind of safety measures are needed to protect patients.

At the center of the debate: state and federal laws limit prescribing to licensed medical professionals. Proponents say those laws, which have underwritten American medicine for over 100 years, should be updated to include AI chatbots and other new technologies.

“We have crossed a threshold in terms of giving something that is not human a medical license, whether or not we want to call it that,” said Dr. Eric Bressman of the University of Pennsylvania.

AI cannot practice medicine under current laws

Bressman and other experts say they aren't opposed to AI prescribing. But they say it should have to meet rigorous standards akin to human doctors, who undergo years of testing and training before being licensed to practice medicine.

In Utah, Doctronic was able to launch thanks to a “regulatory sandbox” that allows state officials to waive laws for AI companies offering promising technology.

The refill program is currently overseen by a five-member board of AI specialists, none of whom are doctors, who say they have implemented numerous safeguards. During the program's initial phase, for example, human doctors review all Doctronic refill orders. The company expects to soon transition to fully automated refills.

The head of the state’s medical licensing board says he and his colleagues learned of the program when its January launch was reported in the news. In a March letter to the state, 11 board members called for the program to be halted, citing the risks of automatically renewing medicines that can have side effects or drug interactions.

“We were essentially told: ‘Yes this is going on. And no, you don’t have a say in it,’” said Dr. Alan Smith, a family physician who heads the board but said he was speaking only for himself.

Complicating the picture is the fact that medical technology is traditionally regulated at the federal level, while medical professionals are overseen by states.

Doctronic executives consider their AI part of the state-regulated practice of medicine. But the federal Food and Drug Administration is supposed to oversee AI that directly impacts medical care or decision making, a line that some experts believe Doctronic has crossed.

Some states are clearing the way for AI in healthcare

In an interview, Doctronic’s executives wouldn't say whether they have sought permission from the FDA.

“Our goal here is really just to meet patients where they need healthcare,” said Dr. Adam Oskowitz, who co-founded the company with a tech industry entrepreneur. “We try not to get too deep into the weeds on the regulatory side.”

In Utah, residents can visit a Doctronic website built for the refill program. After confirming their identity, the AI chatbot asks users about their prescriptions and medical history, verifying that they have a valid prescription by tapping into a national pharmacy database. If there are no issues, the AI can renew the prescription and send it to a local pharmacy. If the request requires more attention, the chatbot transfers the patient to a doctor who works for Doctronic’s telehealth service.

Oskowitz envisions a future where many routine medical tasks, including ordering tests and analyzing results, can be offloaded to Doctronic, allowing doctors to manage thousands more patients than they can today.

Other states are also waiving rules for AI, including Texas and Wyoming.

Meanwhile, lawmakers in Iowa, Idaho and elsewhere have introduced legislation to formally license AI medical services. Many of the bills are based on a template from the nonprofit Cicero Institute, a pro-AI think tank founded by Joe Lonsdale, co-founder of the artificial intelligence software company Palantir.

Pushback against medical AI mainly stems from the economic fears of doctors and other health workers, says Cicero’s director for health policy.

“Whoever goes first is going to take the slings and arrows because there’s economic interests, concerns about the workforce and what that’s going to mean for jobs,” said Cicero's Adam Meier.

Doctors see potential risks to AI prescription refills

Smith, the medical board chair, says the risks to patients are real. He points out that Doctronic’s list of 190 refillable medications includes blood thinners, which can become dangerous if patients develop stomach ulcers or other conditions that cause internal bleeding.

“Many times when I see people after six months I find that their medical history or situation has changed,” Smith said. “Just because something was prescribed before does not mean it’s appropriate now.”

The American Medical Association has voiced similar concerns, warning that “prescription renewals aren’t routine checkboxes.”

Zach Boyd, who heads Utah's AI office, said Doctronic has thus far been overly cautious, often elevating uncontroversial decisions to doctors. In response to safety concerns, several medications have been removed from the list eligible for refills, including a drug for irregular heartbeats.

Utah has released some initial data on the program and Doctronic plans to publish peer-reviewed studies later this year. Currently the only publication about its technology is a paper written by company scientists that was not independently reviewed.

The study looked at whether Doctronic could correctly diagnose medical conditions based on records from 500 telehealth consultations. In the study, Doctronic's diagnoses matched that of human doctors 80% of the time.

The FDA is taking a hands-off approach

Bressman says Utah should have demanded data on prescription refills up front, not after Doctronic was up and running.

“Mostly they're accepting the company’s word on good faith that they’re up to the task,” he said.

The current approach to AI mirrors the haphazard medical standards of the early 20th century, Bressman says, before medical schools, medical boards and other authorities agreed on national benchmarks for training and licensing.

National guidelines on medical technology would typically come from the FDA, but the agency has indicated it plans to take a hand-off approach, at least under the current administration.

An FDA spokesperson said the agency has not authorized any AI chatbots but “is committed to encouraging medical innovation and helping bring promising new technologies to patients, while keeping safety at the center of every decision.”

For now, Doctronic and other companies are likely to expand across states with different regulatory approaches.

“Companies may benefit in the short term by expanding their business models and kind of having the technology go beyond the evidence,” says Daniel Aaron of University of Utah's law school. “But in the long-term, I think they risk compromising public trust and fueling backlash.”

The Associated Press Health and Science Department receives support from the Howard Hughes Medical Institute’s Department of Science Education and the Robert Wood Johnson Foundation. The AP is solely responsible for all content.