DEI Prompts Lead Medical Language Models to Invent Patient Demographics, Preprint Finds

A newly posted research paper suggests that a simple change in wording can push medical language models to make up patient demographics that were never stated in the case. In the study, adding a one-sentence diversity, equity and inclusion prompt to medical questions made all 47 tested models far more likely to inject details such as race or socioeconomic status, and in a smaller subset of responses those invented details changed the model’s answer.

The paper, “Demographic Injection in Medical Language Models under Diversity, Equity, and Inclusion Prompts,” was posted Aug. 15 on arXiv by Arizona State University researchers Diego Mardian and Frank Liu in Tempe, Arizona. It has not been peer-reviewed. The findings matter because they point to a prompt-design problem in medical AI: wording meant to encourage attention to equity may also trigger models to rewrite the patient described in a question.

The authors said they tested 47 language models across four medical multiple-choice benchmarks: MedQA, MedMCQA, MMLU-medical and PubMedQA. In all, they analyzed 376,000 matched responses. Their central finding was that appending a one-sentence DEI prompt raised the share of responses that added an unstated patient demographic from 0.7% at baseline to 33.1%, a 47-fold increase. The paper says the effect appeared in all 47 models tested, including frontier proprietary, budget proprietary, open-source and medical fine-tuned systems. The increase was not simply a result of making prompts longer, the authors said: demographic injection was 18 times higher than in a length-matched neutral control, with one reported comparison showing p = 1.4×10^-14. “We call this demographic injection,” the abstract says.

The paper defines that behavior as adding race, socioeconomic status, sex or other demographic traits that were not present in the original question. According to the authors, injected attributes were dominated by race or ethnicity, at 61%, and socioeconomic status, at 52%. But the paper also draws an important distinction. About 91% of injections were general-population statements rather than direct claims about the specific patient in the vignette, and 68% were factually correct epidemiologic statements that did not change the answer. The more safety-relevant subset was smaller: among responses with DEI prompts, 0.71% attached a demographic to the specific patient, 2.4% changed the selected answer, and 0.25% did both. When an invented demographic changed the recommended option, the paper says it did so 99.8% toward the incorrect answer.

Prompt wording also mattered. The authors reported that different DEI-style phrasings produced injection rates ranging from 14% to 56%. That matters because equity-focused instructions are not an artificial edge case. In clinical AI, guidance increasingly encourages models to consider diversity, inclusion and social determinants of health, meaning the tested prompt style is plausible in real prompting workflows. Prior research has already documented fairness and demographic-sensitivity problems in medical AI. What this paper adds is a more specific claim: that equity-related wording can itself induce a model to invent who the patient is.

The study has notable limits. It is a short preprint, not a regulatory evaluation or a test of real clinical deployment. It relies on multiple-choice benchmarks rather than real-world clinical workflows. Some labels depend on single-annotator gold labels, and the authors’ explanation for why the effect occurs remains a hypothesis rather than a proven mechanism. The paper also explicitly cautions against treating the flagged outputs as medical recommendations: “Flagged outputs are treated as model errors under study, not clinical guidance.” As of Aug. 18, there was no prominent press coverage, institutional press release or public vendor response tied specifically to the preprint, which should be read as an early research finding rather than settled evidence.