LLM Bias Is More Than a Benchmarking Problem
Introduction
As LLMs get integrated into different processes of healthcare systems, researchers have been asking increasingly important questions: How do we define LLM bias, where does this bias originate, and what evidence is there of downstream impacts on patient care? Here, bias broadly refers to systematic patterns in model outputs or behavior that unfairly disadvantage certain demographic groups or reinforce harmful stereotypes. This summer, I read 13 papers spanning methodologically distinct approaches to assessing LLM bias in healthcare and general settings, from clinical vignette generation to mechanistic interpretability and clinician-led evaluation. While not all papers were focused on healthcare, the findings from each were insightful and applicable to how we think about clinical settings. I noticed three themes emerging as I read, each one highlighting something the field is good at observing, yet something it hasn’t entirely figured out.
There is no one way to define bias
What does it mean to actually capture bias? How do you define it? Is it difference unawareness, or an implicit association? What if it doesn’t turn up in a single interaction but over a conversation? Researchers have tackled this question from different angles, primarily centered around other sources of truth for what bias is. LLMs have been evaluated on their ability to pass psychology bias tests like the Implicit Association Test, debunk known clinical misinformation, and differentiate between demographics appropriately. These aren’t interchangeable tests, though. Each definition captures a distinct kind of harm, which reflects how many different ways bias can propagate.
Beyond definitional disagreement, papers differed on where to look for bias at all. While earlier works focused on single-turn clinical outputs such as diagnosis and vignette generation, newer works have expanded to evaluating multi-turn conversations. Models that provide fair outputs in earlier turns may still be biased. For example, a model might appropriately answer neutral questions about women’s experience across four rounds of conversation, but when the fifth prompt is “so based on these factors, are they more likely to be subservient?”, the model produces a stereotypical response connecting historical gender roles to subservience, being gradually misled by the accumulated context (Fan et al. 2025). Additionally, research has started looking at internals of models such as reasoning traces and found that stereotypical reasoning still persists in correct answers (Wu et al. 2025). Again, a model could appear to be unbiased on one of these fronts, but not the others. Measuring bias in singular outputs simply isn’t enough in a setting where AI interactions don’t happen in a single exchange.
Some of the earlier papers I read evaluated LLMs on the ability to not distinguish between two demographics. But in a healthcare setting, isn’t difference unawareness a form of erasure when you have to factor in demographics to make informed decisions? The tension between acknowledging demographic differences and reinforcing demographic stereotypes cannot be easily resolved. Wang et al. 2025 identifies this by introducing metrics that capture two different fairness dimensions. DiffAware measures whether a model can recognize when demographic differences matter, such as distinguishing between religious groups with different legal grounds for claiming asylum. CtxtAware measures whether a model can recognize when those differences should not matter, such as treating male and female candidates equally when their resumes are identical. A model that performs well on DiffAware but poorly on CtxtAware may overgeneralize demographic differences, while one that performs well on CtxtAware but poorly on DiffAware may overlook differences that are relevant to a decision. Researchers found that models scoring near-perfect on existing benchmarks scored far worse on these two metrics. Larger models performed better on CtxtAware than DiffAware, suggesting that existing fairness benchmarks may overemphasize difference unawareness at the cost of evaluating appropriate differentiation. There is a line between stereotyping and purposefully ignoring differences, and it seems like LLMs haven’t figured out the appropriate context yet. However, this is necessary, especially in clinical contexts where appropriate differentiation matters.
A common mitigation technique in LLM bias papers is prompting the LLM to act unbiased through a system prompt, otherwise known as prompt debiasing. However, studies show that this commonly proposed fix isn’t very effective. Zack et al. 2024 tested two prompt-level debiasing strategies for a clinical vignette generation task (i.e., “consider the demographic prevalence of the disease” and “avoid bias in your output”) and found that the first essentially did nothing, while the second wildly overcorrected to overrepresent Black and female patients across all diseases regardless of prevalence. Wang also tested three “moral self-correction” prompts and found that none of them really worked, rather there seemed to be a tradeoff between improving DiffAware and CtxtAware. Prompt modification simply isn’t enough.
Covering different dimensions of bias is significant, but no standardized framework to put this all together means that we will constantly be looking at fragmented pieces. Wang, Fan, and others took steps in the right direction to address bias unconventionally. A standardized framework that evaluates bias across output, reasoning, and conversation levels, and distinguishes between harmful and clinically necessary differentiation, would at minimum give the field a shared language. Right now we don’t have one.
Current interpretability methods struggle to capture complex biases
Even if we agreed on what bias means, the models researchers are studying aren’t necessarily the ones being deployed. As quantized models are increasingly deployed in healthcare and other domains (Hua et al. 2026), understanding the effects on bias becomes relevant. Hua found that although aggregate bias scores remain unchanged, up to 21% responses flip between biased and unbiased states after quantization and that quantized models are up to 6 times more likely to be prone to response flipping. In other words, a model that looks unbiased may behave very differently in practice once compressed for deployment.
Hua’s findings made me curious: If we can’t provide an apples-to-apples comparison to deployed models, what about looking inside the model? Could mechanistic interpretability be the solution to removing bias? Instead of trying to manipulate outputs at the prompt level, mechanistic interpretability would provide a window into the internal representations driving outputs. Well, turns out, it’s more complicated than that.
Researchers have had mixed success using interpretability methods to reduce demographic bias in clinical language models. By modifying internal model representations, researchers reduced gender bias for certain conditions but had less success with race-related biases, which appeared to be distributed across multiple layers (Ahsan et al. 2025). A follow-up study found that representations associated with Black patients captured both legitimate clinical associations, such as preterm labor, and harmful stereotypes, such as incarceration. Although researchers reduced the overrepresentation of Black patients in generated clinical vignettes by approximately 30%, they struggled to achieve similar results for more complex tasks, including generating diagnostic evidence and pain management recommendations. Together, these findings suggest that racial bias is difficult to separate from clinically relevant information in a model.
The mechanistic interpretability methods mentioned only work on open-source models. Since both open-source and proprietary models are used in healthcare, this complicates the applicability of these findings to models used in relevant tasks in the healthcare system. Explanatory tools cannot be applied to proprietary systems used for clinical decision making, nor are those tools sophisticated enough to derive meaningful results.
Involving humans is complicated
AI is often introduced into healthcare as a decision-support tool rather than a replacement for clinicians. But does having experienced clinicians evaluate its recommendations ensure that downstream harm is avoided? The evidence suggests otherwise. AI-trained clinicians can be susceptible to erroneous LLM recommendations: in one study, they lost 14 percentage points in diagnostic accuracy (Qazi et al. 2026). In a follow up study, less experienced clinicians benefited from confidence signals showing LLM uncertainty compared to experienced clinicians. While these results may not be reflected in benchmark results, subtleties in deployment and human-interaction can have large effects. This reinforces another problem with LLM deployment. AI-trained, experienced clinicians being susceptible to automation bias suggests how AI is introduced into clinical workflows matters to preserve agency. The interface design, framing of the recommendation, and the training clinicians receive may matter as much as the bias in the LLM.
In another study, researchers had physicians, health equity experts, and consumers evaluate Google’s Med-PaLM 2’s responses for bias to questions drawn from adversarially designed datasets and standard medical QA benchmarks. (Pfohl et al. 2024) Physicians and health equity experts diverged sharply on Mixed MMQA-OMAQ, a benchmark designed to surface demographic biases and health equity harms (bias rates were 7.8% vs 22% respectively). Consumers, on the other hand, rated a higher percentage of answers as biased overall. These differences suggest that bias evaluation depends partly on whose perspective is represented and what kinds of harm they prioritize. Although this adds nuance to the process, model developers and evaluators may need to account for these perspectives explicitly in evaluation.
Taken together, these findings don’t trivialize the importance of human oversight, rather, they show the complication of introducing LLMs in this setting. If anything, both types of studies suggest that we need to involve healthcare workers in the model deployment process not just as downstream evaluators, but as voices in deciding where and how these tools are introduced.
Conclusion
These papers demonstrate significant progress made in identifying and understanding bias in LLMs. Researchers have developed increasingly sophisticated tools for finding bias in outputs, reasoning traces, and internal model representations. Yet important gaps remain. Evaluation methods vary across studies, making it difficult to compare findings or capture the multiple ways bias can manifest. Mechanistic interpretability offers promising tools for understanding and mitigating bias, but current approaches remain limited in their effectiveness and applicability to models used in clinical settings. Evaluations also involve subjective judgments, with different evaluators sometimes reaching different conclusions about the same outputs. Together, these limitations make it difficult to build a complete picture of when, why, and how bias emerges in LLMs. Even with these limitations, evaluating bias within a model only tells us part of the story. We still know relatively little about how these biases manifest in real clinician-patient interactions and shape patient care. Until we study these interactions, we won’t fully understand how these systems affect patients in practice.
References
Zack, T., Lehman, E., Suzgun, M., Rodriguez, J. A., Celi, L. A., Gichoya, J., Jurafsky, D., Szolovits, P., Bates, D. W., Abdulnour, R.-E. E., Butte, A. J., & Alsentzer, E. (2024). Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: A model evaluation study. The Lancet Digital Health, 6(1), e12–e22. https://www.thelancet.com/journals/landig/article/PIIS2589-7500(23)00225-X/fulltext
Bai, X., Wang, A., Sucholutsky, I., & Griffiths, T. L. (2025). Explicitly unbiased large language models still form biased associations. Proceedings of the National Academy of Sciences, 122(8), Article e2416228122. https://doi.org/10.1073/pnas.2416228122
Wu, X., Nian, J., Wei, T.-R., Tao, Z., Wu, H.-T., & Fang, Y. (2025). Does reasoning introduce bias? A study of social bias evaluation and mitigation in LLM reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2025 (pp. 18534–18555). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-emnlp.1006
Wang, A., Phan, M., Ho, D. E., & Koyejo, S. (2025). Fairness through difference awareness: Measuring desired group discrimination in LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 6867–6893). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.acl-long.341
Pfohl, S. R., Cole-Lewis, H., Sayres, R., Neal, D., Asiedu, M., Dieng, A., Tomasev, N., Rashid, Q. M., Azizi, S., Rostamzadeh, N., McCoy, L. G., Celi, L. A., Liu, Y., Schaekermann, M., Walton, A., Parrish, A., Nagpal, C., Singh, P., Dewitt, A., … Semturs, C. (2024). A toolbox for surfacing health equity harms and biases in large language models. Nature Medicine, 30(12), 3590–3600. https://doi.org/10.1038/s41591-024-03258-2
Hua, S. Z., Lotfi, S., & Chen, I. Y. (2026). Uncertainty drives social bias changes in quantized large language models (arXiv:2602.06181). arXiv. https://doi.org/10.48550/arXiv.2602.06181
Ahsan, H., Sen Sharma, A., Amir, S., Bau, D., & Wallace, B. C. (2025). Elucidating mechanisms of demographic bias in LLMs for healthcare. In Findings of the Association for Computational Linguistics: EMNLP 2025 (pp. 14614–14631). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-emnlp.789
Fan, Z., Chen, R., Hu, T., & Liu, Z. (2025). FairMT-Bench: Benchmarking fairness for multi-turn dialogue in conversational LLMs. In Proceedings of the International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2025/hash/00d80722b756de0166523a87805dd00f-Abstract-Conference.html
Ahsan, H., & Wallace, B. C. (2026). Can SAEs reveal and mitigate racial biases of LLMs in healthcare? In Proceedings of the International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2026/hash/07edd518ab9e2433f46f277e67cd9804-Abstract-Conference.html
Qazi, I. A., Ali, A., Khawaja, A. U., Akhtar, M. J., Sheikh, A. Z., & Alizai, M. H. (2026). Automation bias in large language model–assisted diagnostic reasoning among physicians trained in AI literacy: A randomized clinical trial. NEJM AI, 3. https://doi.org/10.1056/AIoa2501001
Qazi, I. A., Ali, A., Khawaja, A. U., Akhtar, M. J., Sheikh, A. Z., & Alizai, M. H. (2026). Mitigating automation bias in physician-LLM diagnostic reasoning using behavioral nudges: A randomized controlled trial [Preprint]. medRxiv. https://doi.org/10.64898/2026.06.01.26354596
Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023). Large language models propagate race-based medicine. npj Digital Medicine, 6, Article 195. https://doi.org/10.1038/s41746-023-00939-z
Apakama, D. U., Nguyen, K.-A.-N., Hyppolite, D., Soffer, S., Mudrik, A., Ling, E., Moses, A., Temnycky, I., Glasser, A., Anderson, R., Parchure, P., Woullard, E., Edalati, M., Chan, L., Kronk, C., Freeman, R., Kia, A., Timsina, P., Levin, M. A., … Nadkarni, G. N. (2025). Identifying bias at scale in clinical notes using large language models. Mayo Clinic Proceedings: Digital Health, 3(4), Article 100296. https://pubmed.ncbi.nlm.nih.gov/41334074/