What Apple's health-AI study needs to tell us about trust

Apple and OpenEvidence are studying how people use AI for health questions. A useful answer could leave someone less certain because they understand a limitation they had missed.

If someone finishes a health conversation less sure of an answer, I would ask them to explain what changed. They may have learned why a medical study does not quite fit their circumstances. They may now have a question to take to their doctor that they would not have known to ask before.

Apple announced its study with OpenEvidence on November 15. Participants can ask the app health questions and follow links to medical research in its answers. Apple says it will examine trust, comprehension and everyday health actions. The app is not intended to diagnose or treat a condition or replace a clinician.

The study has room for up to 200,000 US adults aged 21 or older with a compatible Apple Intelligence-enabled iPhone. I would look at who actually joins before applying the results to a wider public. Access to that phone and a willingness to take part are already part of who the researchers will hear from.

Participants can choose whether to share HealthKit information, and Apple says study data will not be used to train AI models. Before sharing, someone also needs to understand what happens to information already collected if they leave the study.

let the person explain the answer

Researchers can ask participants to explain the answer in their own words and describe where they think it applies to them, then compare those explanations with their confidence ratings.

Illustrative responses ยท no study results

The same rating can hide different understanding

Person A

Confidence in the answer
High
Explanation afterward
Can describe a limitation and why it applies.
Next action
Still needs to be observed.

Person B

Confidence in the answer
High
Explanation afterward
Repeats the conclusion without its limitation.
Next action
Still needs to be observed.
These are invented examples of what separate measures could reveal. They do not describe Apple study participants.

The explanation would give the researchers something specific to follow. Did the answer leave the qualification out? Did the person miss it, or understand it and decide it was less relevant to their question? The explanation can help the researchers decide whether to change the app's answer or ask more about how the person used it. Apple's announcement does not describe those measurements.

Andrew Bean and colleagues examined people using language models for health questions in a 2025 preprint. They randomly assigned 1,298 people in the UK to use one of three models or their usual methods, working through up to two of ten hypothetical medical scenarios. People using the models were less successful at identifying relevant conditions than the control group and did not significantly improve at choosing an appropriate course of action.

The models identified conditions substantially better in separate tests with standardized prompts. The researchers found incomplete information and model misinterpretations in some conversations. These were different systems from Apple's app, but the study gives us a reason to examine what happens when the person has to describe the concern and make sense of the response. A model receiving a complete scenario has been given work that a person asking a question still has to do.

follow the explanation into the next decision

A link to a medical paper gives a person somewhere to check the answer. If the study involved a different age group, it could still be relevant, but the person needs to understand why. Asking them to explain the connection would tell the researchers more than counting whether the link opened. The answer also needs to explain an important limitation for someone who never reads the paper.

Apple says it will explore whether the app helps people prepare for a medical visit. If a participant feels better prepared, I would ask what they now plan to discuss with the clinician. The study could then follow whether the person used that question, if its design allows that follow-up. A plan to call and an appointment that happened need to be reported separately, so readers can tell what the study observed.

Related: AI is helping attackers exploit unpatched software examines the work needed to install an urgent software fix.