The Unexpec ed Ha ms o A i icial
In elligence in Heal hca e: Re lec ions on
Fou Real-Wo ld Cases
Ke s in DENECKEa,1 Guille mo LOPEZ-CAMPOSb, Oc a io RIVERA-ROMEROc,
and Elia GABARRONd
a Be n Uni e si y o Applied Sciences, Be n, Swi ze land
bWellcome-Wol son Ins i u e o Expe imen al Medicine, Queen’s Uni e si y Bel as
cDepa men o Elec onic Technology, Uni e sidad de Se illa, Se ille, Spain
dDepa men o Educa ion, ICT and Lea ning, Øs old Uni e si y College, No way
ORCiD ID: Ke s in Denecke h ps://o cid.o g/0000-0001-6691-396X, Guille mo
Lopez-Campos h ps://o cid.o g/0000-0003-3011-0940, Oc a io Ri e a-Rome o
h ps://o cid.o g/0000-0001-7212-9805, Elia Gaba on h ps://o cid.o g/0000-0002-
7188-550X
Abs ac . In oduc ion: Rapid ad ances in A i icial In elligence (AI), especially
wi h la ge language models, p esen bo h oppo uni ies and challenges in heal hca e.
This a icle analyzes eal-wo ld AI- ela ed ha ms in heal hca e. Me hods: We
selec ed ou ecen AI- ela ed inciden s om he AIAAIC Reposi o y. Resul s: The
inciden s discussed include: Whispe ’s ha m ul hallucina ions; UNOS’s algo i hm
delaying ansplan s o black pa ien s; he WHO’s S.A.R.A.H. cha bo p o iding
inaccu a e heal h in o ma ion; and Cha ac e AI’s cha bo p omo ing diso de ed
ea ing among eens. Discussion and conclusion: These inciden s highligh di e se
isks, om misin o ma ion o sa e y conce ns, in ol ing bo h indus y and
ins i u ional p o ide s. The a icle emphasizes he need o sys ema ic epo ing o
AI- ela ed ha ms, conce ns abou secu i y, p i acy, and e hics, and calls o a
cen alized heal h-speci ic da abase o enhance pa ien sa e y and unde s anding.
Keywo ds. A i icial in elligence, Digi al echnology, Digi al heal h in e en ions,
Ad e se e en s, Pa ien sa e y
1. In oduc ion
Rapid ad ances in a i icial in elligence (AI), pa icula ly wi h la ge language models
(LLMs) and gene a i e AI, ha e c ea ed bo h new oppo uni ies and challenges in
heal hca e. These echnologies ha e demons a ed ema kable capabili ies in
unde s anding and gene a ing human language, and ha e p o en highly e ec i e in
na u al language p ocessing (NLP) asks like ansla ion [1], summa iza ion [2],
classi ica ion [3], named en i y ecogni ion, and medical ques ion answe ing [4]. Despi e
hese ecen ad ances, AI has been used in medical in o ma ics o o e hal a cen u y
and has been ex ensi ely used o decades in di e en a eas such as he de elopmen o
1
Co esponding Au ho : Ke s in Denecke. Be n Uni e si y o Applied Sciences, Quellgasse 21, 2502
Biel/Bienne, Swi ze land. Mail: [email p o ec ed].
Heal hca e o he Fu u e 2025
T. Bü kle e al. (Eds.)
© 2025 The Au ho s.
This a icle is published online wi h Open Access by IOS P ess and dis ibu ed unde he e ms
o he C ea i e Commons A ibu ion Non-Comme cial License 4.0 (CC BY-NC 4.0).
doi:10.3233/SHTI250219
55
clinical decision suppo sys ems [5,6]. Along i s way, he de elopmen o such solu ions
has no been exemp ed om challenges and conce ns abou he de elopmen o such
app oaches. A pa adigma ic ecen example o hese isks we e he acial biases in
measu emen s coming om pulse-oxime e s and o he medical de ices [7,8]. While AI
in eg a ion can inc ease e iciency and op imiza ion o heal hca e p ocesses, conce ns
emain ega ding accu acy, egula o y compliance, p i acy and secu i y, human ac o s,
and e hical conside a ions [9].
This esea ch a icle aims o documen and analyze examples o eal-wo ld cases o
inciden s and ha ms in heal hca e linked o he use o AI, emphasizing he c i ical
impo ance o eco ding hese e en s in he scien i ic li e a u e o be e unde s and hei
scope and implica ions, and o guide he de elopmen o s a egies o mi iga ing isks
and p omo ing he sa e adop ion o AI in clinical se ings.
2. Me hods
We andomly selec ed ou ecen AI- ela ed inciden s om he AIAAIC Reposi o y:
wo a ec ing heal hca e p o essionals and wo impac ing gene al popula ion and
child en. AIAAIC is one o he independen ini ia i es ha ocuses on ad oca ing o
anspa ency and openness in AI algo i hms. This ini ia i e main ains a eposi o y whe e
ha ms o AI and algo i hmic sys ems ac oss all sec o s a e eco ded [10]. As o ea ly
2025, i has eco ded 1,904 inciden s since 2008, wi h o e 100 linked o heal hca e.
Table 1 summa izes he co e ed di e se echnologies, isks, asks, and use s in ol ed in
he ou selec ed cases.
Table 1. O e iew o he desc ibed cases
Aspec s Case 1 Case 2 Case 3 Case 4
Technology Speech- o-Tex
Technology
Algo i hm o
p io i iza ion
Cha bo Cha bo
Risk / Sa e y
conce n
Hallucina ions, Risk
o pa ien sa e y,
Risk o da a in eg i y
in heal h eco ds
Delayed ansplan s,
pa ien sa e y
Ou da ed
in o ma ion,
inaccu a e
in o ma ion
"Coach"
ano exia-like
beha io s
Task Clinical
documen a io
n
Decision-making Gene al heal h
ad ice
Cha
Use Heal h p o essionals Heal h p o essionals Gene al popula ion Teenage s
3. Resul s
3.1. Case 1: AI ansc ip ions as a isk o pa ien sa e y and da a in eg i y [11,12]
Whispe , an au oma ic speech ecogni ion sys em ained on 680,000 hou s o
mul ilingual da a, has been ound o gene a e alse ex , some imes p oducing en i e
sen ences ha we e no p esen in he o iginal audio [13]. These "hallucina ions" can
in ol e ha m ul con en , such as acis commen s, iolen he o ic, and ab ica ed
medical ea men s, like a non-exis en d ug called "hype ac i a ed an ibio ics." A s udy
in ol ing 13,140 audio segmen s ound ha 1.4% con ained hallucina ions, wi h nea ly
40% being ha m ul o conce ning [14]. While no di ec pa ien ha m has been epo ed,
inaccu a e clinical ansc ip s pose isks o pa ien sa e y. Al hough Whispe ansc ibed
K. Denecke e al. / The Unexpec ed Ha ms o A i icial In elligence in Heal hca e56
spoken con en co ec ly, i added alse in o ma ion, including iolence, inaccu a e
associa ions, and alse au ho i y [14]. Despi e OpenAI’s wa nings agains using Whispe
in high- isk a eas, i is being adop ed in heal hca e, aising conce ns abou pa ien sa e y,
medical eco d in eg i y, and con iden iali y.
3.2. Case 2: Algo i hm delays ansplan s o black pa ien s and you h [15,16]
Se e al inciden s highligh biases in algo i hms used o p io i ize o gan ansplan
pa ien s. In Ap il 2023, he UNOS’s UNe algo i hm in he U.S. was ound o un ai ly
delay kidney ansplan s o Black pa ien s by o e es ima ing hei kidney unc ion,
leading o longe wai imes [15]. Simila ly, he T ansplan Bene i Sco e algo i hm in
he UK, in oduced in 2018, assigned lowe sco es o younge pa ien s, educing hei
chances o ecei ing a li e ansplan [16]. These inciden s highligh he impo ance o
analyzing biases in AI algo i hms used in heal hca e. Heal hca e p o essionals mus be
awa e o hese biases o p e en disc imina o y ou comes. In one case, his bias esul ed
in a pa ien wai ing o e i e yea s o a kidney ansplan .
3.3. Case 3. WHO cha bo p o ides inaccu a e heal h in o ma ion [17]
In Ap il 2024 he Wo ld Heal h O ganiza ion (WHO) eleased S.A.R.A.H. [18], a digi al
heal h p omo e based on Cha GPT3.5, designed o p o ide guidance on opics such as
men al heal h, heal hy ea ing o qui ing smoking amids a g owing sho age o
heal hca e wo ke s. Howe e , a media epo sho ly a e he o icial elease o he ool
ha he sys em ailed o p o ide upda ed and accu a e in o ma ion. The WHO
acknowledged hese limi a ions, no ing ha S.A.R.A.H. is s ill a wo k in p og ess and
o en di ec s use s o i s websi e o heal hca e p o ide s. The inciden highligh s conce ns
abou he accu acy and imeliness o AI in heal hca e. S.A.R.A.H. includes a disclaime
ha i s esponses do no e lec WHO's iews and a e no gua an eed o be accu a e.
Simila issues p e iously led o he shu down o an ea ing diso de suppo cha bo [19].
As o his w i ing, no u he s udies o upda es on S.A.R.A.H. ha e been ound, and i
emains unclea whe he i s pe o mance has imp o ed.
3.4. Case 4: Cha ac e AI encou ages kids o engage in diso de ed ea ing [20]
Cha ac e AI, a pla o m hos ing cha bo pe sonas [21], aced media exposu e a e some
o i s cha bo s, like "4n4 Coach" (a wis on "ana", he online nickname o ano exia),
p omo ed diso de ed ea ing beha io s among eens. These bo s encou aged dange ously
low-calo ie die s (e.g., 900–1,200 calo ies daily), meal skipping, and excessi e exe cise,
engaging nea ly 14,000 use s [20] and highligh ing lapses in con en mode a ion, age
es ic ions, and e hical o e sigh . While no di ec ha m has been p o en, exposu e o
p o-ano exia con en can nega i ely impac adolescen s' body image and ea ing
beha io s [22,23]. I is wo h men ioning ha a he ime o w i ing his a icle, he "4n4
Coach" no longe appea s in sea ch esul s. Howe e o he p o-ano exia bo s emain
ac i e, some wi h o e 1,000 use s. This inciden highligh s he isks o un egula ed AI,
especially o ulne able you h, aising conce ns abou ea ing diso de s and men al
heal h. As o ea ly 2025, o ou knowledge, he e a e no publica ions indexed in a leading
heal h li e a u e da abase (e.g., PubMed) e e encing his case.
K. Denecke e al. / The Unexpec ed Ha ms o A i icial In elligence in Heal hca e 57
4. Discussion
The desc ibed AI echnologies a ge di e en audiences, om he gene al public (WHO
cha bo ), eens (Cha ac e AI), o heal hca e p o essionals ( ansplan algo i hms, speech-
o- ex ools). The isks also a y, wi h one case di ec ly a ec ing pa ien sa e y and
o he s posing po en ial ha m. These cases in ol e bo h indus y and ins i u ional
p o ide s, bu only one had scien i ic li e a u e suppo [14].
The g owing use o AI in heal h d i es bo h esea ch and indus y. Resea ch
p io i izes clinical e ec i eness and bes p ac ices [24], while indus y ope a es in bo h
egula ed and un egula ed spaces, whe e isks may go un epo ed. Fu he mo e, no
cen alized da abase exis s o AI- ela ed heal hca e inciden s, and exis ing eposi o ies,
such as AIAAIC [10], AI Inciden Da abase [25] and he OECD AI Inciden Moni o
[26] ely on olun a y epo ing. In he USA, AI- ela ed medical de ice issues may be
ound in he FDA’s MAUDE da abase [27]. Howe e , he absence o a dedica ed heal h-
speci ic da abase limi s isk unde s anding, a ec ing pa ien sa e y and public us .
Expe s highligh conce ns ega ding misin o ma ion, secu i y, p i acy, e hics, and
liabili y [9,28-31]. The p esen ed eal-wo ld cases align wi h hese conce ns: Whispe ’s
hallucina ions (case 1) in ol e misin o ma ion, biased ansplan algo i hms (case 2)
ein o ce disc imina ion, and cases 3 and 4 demons a e AI- ela ed sa e y isks. Despi e
wa nings om AI de elope s, heal hca e ins i u ions con inue adop ing AI o add ess
s a sho ages and s eamline p ocesses, o en o e looking isks. Wi hou egula ions,
his end is likely o wo sen. To mi iga e ha m, WHO has issued AI e hics and
go e nance guidance [32], emphasizing pa icipa o y design, isk p edic ion, and
egula o y en o cemen . Scien i ic documen a ion o eal-wo ld AI inciden s is c ucial
o anspa ency and esponsible AI in eg a ion [33].
This s udy has se e al limi a ions. The cases we e andomly selec ed a he han
using a sys ema ic me hodology, limi ing he gene alizabili y and ep esen a i eness o
he indings. While ou goal was o highligh AI- ela ed ha ms in heal hca e and
encou age sys ema ic epo ing, his app oach may no cap u e he ull scope o
equency o such inciden s. Addi ionally, eliance on anecdo al examples p e en s
quan i ying he p e alence o se e i y o hese ha ms, emphasizing he need o u u e
esea ch wi h mo e igo ous me hods. Fu u e esea ch could sys ema ically analyze
documen ed inciden s o ha m ul AI applica ions in heal hca e a ailable in eposi o ies.
5. Conclusion
The eal-wo ld cases p esen ed in his pape highligh he signi ican isks and e hical
challenges associa ed wi h he use o AI in heal hca e, including ansc ip ion
hallucina ions, biased ansplan algo i hms, inaccu a e heal h in o ma ion, and he
p omo ion o diso de ed ea ing. These inciden s, o en unde epo ed in scien i ic
li e a u e, unde sco e he u gen need o moni o ing and sys ema ic epo ing o AI-
ela ed ha ms, while emphasizing he impo ance o ans e ing knowledge om non-
scien i ic media o he scien i ic communi y o add ess hese challenges e ec i ely.
K. Denecke e al. / The Unexpec ed Ha ms o A i icial In elligence in Heal hca e58
Re e ences
[1] Wang W, Yang Z, Gao Y, e al. T ans o me -Based Di ec Hidden Ma ko Model o Machine T ansla ion.
P oceedingso he 59 h Annual Mee ing o he Associa ion o Compu a ional Linguis ics and he 11 h
In e na ional Join Con e ence on Na u al Language P ocessing: S uden Resea ch Wo kshop, Online:
Associa ion o Compu a ional Linguis ics. 2021:23-32.
[2] Mo o G, Ragazzi L, Valgimigli L, e al. E icien Memo y-Enhanced T ans o me o Long-Documen
Summa iza ion in Low-Resou ce Regimes. Senso s. 2023;23:3542.
[3] Dai X, Chalkidis I, Da kne S, e al. Re isi ing T ans o me -based Models o Long Documen
Classi ica ion. In Findings o he Associa ion o Compu a ional Linguis ics: EMNLP 2022. Abu Dhabi,
Uni ed A ab Emi a es Associa ion o Compu a ional Linguis ics. 2022:7212-7230.
[4] Yang X, Pou Neja ian N, Shin HC, e al. Ga o on: A la ge language model o clinical na u al language
p ocessing. MedRxi . 2022:2022-02.
[5] Sho li e EH, Axline SG, Buchanan BG, e al. An a i icial in elligence p og am o ad ise physicians
ega ding an imic obial he apy. Compu Biomed Res. 1973;6(6):544-60.
[6] Hand DJ. A i icial in elligence and medicine: discussion pape . J R Soc Med. 1987;80(9):563-5.
[7] Hidalgo DC, Olusanya O, Ha lan E. C i ical ca e ainees call o pulse oxime y e o m. Lance Respi
Med. 2021;9(4):e37.
[8] Valbuena VSM, Me chan RM, Hough CL. Racial and E hnic Bias in Pulse Oxime y and Clinical
Ou comes. JAMA In e n Med. 2022;182(7):699-700.
[9] Denecke K, May R, Ri e a-Rome o O. T ans o me Models in Heal hca e: A Su ey and Thema ic
Analysis o Po en ials, Sho comings and Risks. J Med Sys . 2024;48(1):23.
[10] AIAAIC (AI AaAIaC. AIAAIC 2024. A ailable om: h ps://www.aiaaic.o g/
[11] AIAAIC. S udy: Whispe AI ansc ip ion se ice in en s medical ea men s 2024. A ailable om:
h ps://www.aiaaic.o g/aiaaic- eposi o y/ai-algo i hmic-and-au oma ion-inciden s/s udy-whispe -ai-
ansc ip ion-in en s-medical- ea men s
[12] AIAAIC. S udy: Whispe AI speech ecogni ion c ea es iolen hallucina ions 2024. A ailable om:
h ps://www.aiaaic.o g/aiaaic- eposi o y/ai-algo i hmic-and-au oma ion-inciden s/s udy-whispe -ai-
speech- ecogni ion-c ea es- iolen -hallucina ions
[13] Bu ke G, Schellmann H. Resea che s say an AI-powe ed ansc ip ion ool used in hospi als in en s hings
no one e e said 2024. A ailable om: h ps://apnews.com/a icle/ai-a i icial-in elligence-heal h-
business-90020cd 5 a16c79ca2e5b6c4c9bbb14
[14] Koenecke A, Seo A, Choi G, e al. Ca eless Whispe : Speech- o-Tex Hallucina ion Ha ms. P oceedings
o he 2024 ACM Con e ence on Fai ness, Accoun abili y, and T anspa ency (FAccT '24) Associa ion o
Compu ing Machine y, New Yo k, NY, USA. 2024:1672-1681.
[15] AIAAIC. Black man sues UNOS o e kidney ansplan algo i hm acial bias 2023. A ailable om:
h ps://www.aiaaic.o g/aiaaic- eposi o y/ai-algo i hmic-and-au oma ion-inciden s/black-man-sues-unos-
o e -kidney- ansplan -algo i hm- acial-bias
[16] AIAAIC. Algo i hm delays young peoples' li e ansplan s 2023. A ailable om:
h ps://www.aiaaic.o g/aiaaic- eposi o y/ai-algo i hmic-and-au oma ion-inciden s/algo i hm-delays-
young-peoples-li e - ansplan s
[17] AIAAIC. WHO cha bo p o ides inaccu a e heal h in o ma ion 2024. A ailable om:
h ps://www.aiaaic.o g/aiaaic- eposi o y/ai-algo i hmic-and-au oma ion-inciden s/who-cha bo -p o ides-
inaccu a e-heal h-in o ma ion
[18] Wo ld Heal h O ganiza ion. S.A.R.A.H, a Sma AI Resou ce Assis an o Heal h 2024. A ailable om:
h ps://www.who.in /campaigns/s-a- -a-h
[19] Wells K. An ea ing diso de s cha bo o e ed die ing ad ice, aising ea s abou AI in heal h 2023.
A ailable om: h ps://www.np .o g/sec ions/heal h-sho s/2023/06/08/1180838096/an-ea ing-diso de s-
cha bo -o e ed-die ing-ad ice- aising- ea s-abou -ai-in-hea
[20] AIAAIC. Cha ac e AI encou ages kids o engage in diso de ed ea ing 2024. A ailable om:
h ps://www.aiaaic.o g/aiaaic- eposi o y/ai-algo i hmic-and-au oma ion-inciden s/cha ac e -ai-
encou ages-kids- o-engage-in-diso de ed-ea ing
[21] Cha ac e Technologies I. h ps://cha ac e .ai/ 2024. A ailable om: h ps://cha ac e .ai/
[22] Men o C, Sil es i MC, Musca ello MRA, e al. Psychological Impac o P o-Ano exia and P o-Ea ing
Diso de Websi es on Adolescen Females: A Sys ema ic Re iew. In J En i on Res Public Heal h.
2021;18(4).
[23] Rodge s RF, Lowy AS, Halpe in DM, e al. A Me a-Analysis Examining he In luence o P o-Ea ing
Diso de Websi es on Body Image and Ea ing Pa hology. Eu Ea Diso d Re . 2016;24(1):3-8.
[24] Embi PJ. Algo i hmo igilance-Ad ancing Me hods o Analyze and Moni o A i icial In elligence-D i en
Heal h Ca e o E ec i eness and Equi y. JAMA Ne w Open. 2021;4(4):e214622.
[25] AI Inciden Da abase. New Inciden Repo 2024. A ailable om: h ps://inciden da abase.ai/apps/submi /
K. Denecke e al. / The Unexpec ed Ha ms o A i icial In elligence in Heal hca e 59
[26] OECD.AI Policy Obse a o y. G7 epo ing amewo k – Hi oshima AI P ocess (HAIP) in e na ional code
o conduc o o ganiza ions de eloping ad anced AI sys ems 2025. A ailable om:
h ps:// anspa ency.oecd.ai/
[27] FDA U.S. Food & D ug Adminis a ion. Manu ac u e and Use Facili y De ice Expe ience (MAUDE)
Da abase 2025. A ailable om: h ps://www.accessda a. da.go /sc ip s/cd h/c docs/c maude/sea ch.c m
[28] Denecke K, May R, Ri e a Rome o O. Po en ial o La ge Language Models in Heal h Ca e: Delphi S udy.
J Med In e ne Res. 2024;26:e52399.
[29] Gua dado S, Mynolopoulou V, Bansi J, e al. An explo a o y s udy on he oppo uni ies and use ulness o
pa ien -gene a ed heal h da a as a ool in he ca e o people wi h Mul iple Scle osis. Me hods o
In o ma ion in Medicine. 2023.
[30] Rani S, Kuma i A, Ekka SC, e al. Pe cep ion o Medical S uden s and Facul y Rega ding he Use o
A i icial In elligence (AI) in Medical Educa ion: A C oss-Sec ional S udy. Cu eus. 2025;17(1):e77514.
[31] S oud AM, Cu is SH, Wei IB, e al. Physician Pe spec i es on he Po en ial Bene i s and Risks o
Applying A i icial In elligence in Psychia ic Medicine: Quali a i e S udy. JMIR Men Heal h.
2025;12:e64414.
[32] Wo ld Heal h O ganiza ion. E hics and go e nance o a i icial in elligence o heal h: Guidance on la ge
mul i-modal models 2024. A ailable om: h ps://www.who.in /publica ions/i/i em/9789240084759
[33] Liu F, Panagio akos D. Real-wo ld da a: a b ie e iew o he me hods, applica ions, challenges and
oppo uni ies. BMC Med Res Me hodol. 2022;22(1):287.
K. Denecke e al. / The Unexpec ed Ha ms o A i icial In elligence in Heal hca e60
