Exploring image-based AI methods for analyzing timelapse EmbryoScope data

id194852


EXPLORING IMAGE-BASED AI METHODS FOR
ANALYZING TIMELAPSE EMBRYOSCOPE DATA
LUKAS MEGGLE
Thesis supe iso
DARIOGARCÍAGASULLA(BARCELONASUPERCOMPUTINGCENTER-CENTRONACIONALDE
SUPERCOMPUTACION)
Tu o :JAVIERBÉJARALONSO(Depa men o Compu e Science)
Deg ee
Mas e 'sDeg eeinA i icialIn elligence
Mas e 's hesis
School o Enginee ing
Uni e si a Ro i a i Vi gili (URV)
Facul y o Ma hema ics
Uni e si a de Ba celona (UB)
Ba celona School o In o ma ics (FIB)
Uni e si a Poli ècnica de Ca alunya (UPC) - Ba celonaTech

Resum
Els a enços en les Tecnologies de Rep oducció Assis ida (ART) han posa de mani es la necessi a
d’eines objec i es i basades en dades pe dona supo a la selecció d’emb ions en la Fecundació In
Vi o (FIV). Aques a esi explo a mè odes d’in el·ligència a i icial basa s en ima ges pe p edi la
o mació de blas ocis s de bona quali a al dia 3, u ili zan dades empo als de emps ob ingudes
d’incubado es Emb yoScope. Es a ecopila i p ep ocessa un conjun de dades pe sonali za amb
10.274 seqüències d’emb ions ano ades, en col·labo ació amb l’Hospi al Clínic de Ba celona, pe a
l’en enamen dels models. Es an a alua di e sos en ocamen s, incloen -hi models adicionals
d’ap enen a ge au omà ic i models d’ap enen a ge p o und, en di e en s modali a s: ima ges indi id-
uals, seqüències de dades empo als de emps i ca ac e ís iques mo ocinè iques usionades. El model
amb millo endimen —basa en E icien Ne B0 i una usió de ca ac e ís iques amb mecanisme
d’a enció— a assoli una p ecisió equilib ada del 77,37% en el conjun de es , men e que un
model comple amen au oma i za basa en ídeo, amb una xa xa LSTM, a a iba a una p ecisió
equilib ada del 75,3%. Els esul a s demos en un al endimen dels models emb iona is, assolin
esul a s d’úl ima gene ació en el conjun de dades disponible i mos an un po encial p ome edo
pe al supo clínic eal en la selecció d’emb ions.
Resumen
Los a ances en las Tecnologías de Rep oducción Asis ida (ART) han sub ayado la necesidad de
he amien as obje i as y basadas en da os pa a apoya la selección de emb iones en la Fecundación In
Vi o (FIV). Es a esis explo a mé odos de in eligencia a i icial basados en imágenes pa a p edeci la
o mación de blas ocis os de buena calidad en el día 3, u ilizando da os de lapso de iempo ob enidos
de incubado as Emb yoScope. Se ecopiló y p ep ocesó un conjun o de da os pe sonalizado con
10.274 secuencias de emb iones ano adas, en colabo ación con el Hospi al Clínic de Ba celona, pa a
el en enamien o de los modelos. Se e alua on múl iples en oques, incluidos modelos adicionales
de ap endizaje au omá ico y modelos de ap endizaje p o undo, en di e en es modalidades: imágenes
indi iduales, secuencias de lapso de iempo y ca ac e ís icas mo ociné icas usionadas. El modelo
con mejo endimien o —basado en E icien Ne B0 y una usión de ca ac e ís icas con a ención—
log ó una p ecisión equilib ada del 77,37% en el conjun o de p ueba, mien as que un modelo
comple amen e au oma izado basado en ideo, que inco po a una ed LSTM, alcanzó una p ecisión
equilib ada del 75,3%. Los esul ados demues an un al o endimien o de los modelos emb iona ios,
alcanzando esul ados de úl ima gene ación en el conjun o de da os disponible y mos ando un
po encial p ome edo pa a el apoyo clínico eal en la selección de emb iones.

Abs ac
Ad ances in Assis ed Rep oduc i e Technology (ART) ha e unde sco ed he need o objec i e,
da a-d i en ools o suppo emb yo selec ion in In Vi o Fe iliza ion (IVF). This hesis explo es
image-based a i icial in elligence me hods o p edic ing good-quali y blas ocys o ma ion a day
3 using ime-lapse da a om Emb yoScope incuba o s. A cus om da ase o 10,274 anno a ed
emb yo sequences, collec ed in collabo a ion wi h Hospi al Clínic de Ba celona, was assembled and
p ep ocessed o model aining. Mul iple app oaches, including adi ional machine lea ning and
deep lea ning models, we e e alua ed ac oss di e en modali ies: single images, ime-lapse sequences,
and used mo phokine ic ea u es. The bes -pe o ming model - based on a E icien Ne B0 and
a en ion-based ea u e usion - achie ed a balanced accu acy o 77.37% on he es se , while a
ully au oma ed ideo-based model inco po a ing an LSTM ne wo k eached a balanced accu acy o
75.3%. The esul s demons a e a s ong pe o mance o he emb yo models, eaching s a e-o - he-a
esul s on he da ase a hand and showing p omising po en ial o eal-wo ld clinical suppo in
emb yo selec ion.
Table o Con en s
Lis o Figu es 3
Lis o Tables 5
Abb e ia ions 7
1 In oduc ion, Mo i a ion and Objec i es 9
2 S a e o he A 11
2.1 Mo phology and Mo phokine ics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 AI Models o Emb yo Assessmen . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.1 Emb yo Labelling, Sco ing and Selec ion . . . . . . . . . . . . . . . . . . . . . 12
2.2.2 Blas ocys P edic ion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3 Da ase 15
3.1 Me ada a .......................................... 15
3.2 Emb yoSampleSelec ion ................................. 17
3.3 Labeling........................................... 18
3.4 Fea u e Ex ac ion and Impu a ion S a egy . . . . . . . . . . . . . . . . . . . . . . . 18
3.5 F ameTimeEx ac ion .................................. 19
3.6 FinalDa ase P epa a ion................................. 20
4 Me hods 21
4.1 MachineLea ningModel.................................. 21
4.2 DeepLea ningModels................................... 22
4.2.1 SingleImageModel ................................ 22
4.2.2 Emb yoFea u eModel .............................. 26
4.2.3 Emb yoVideoModel ............................... 27
5 Resul s 31
5.1 MachineLea ningModel.................................. 31
5.2 DeepLea ningModels................................... 31
5.2.1 Single Image Model Resul s . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.2.2 Emb yoFea u eModel .............................. 34
5.2.3 Emb yoVideoModel ............................... 35
6 Sus ainabili y analysis and e hical implica ions 37
1
PN p onuclea . 15, 18, 55
TE ophec ode m. 11, 41
UPC Uni e si a Poli ècnica de Ca alunya. 10
ViTs Vision T ans o me s. 23
Z-sco e zygo e sco e. 11
8

Chap e 1
In oduc ion, Mo i a ion and
Objec i es
In Vi o Fe iliza ion (IVF) is one o he mos widely applied Assis ed Rep oduc i e Technology
(ART) o indi iduals and couples expe iencing in e ili y. Since i s i s success ul use in 1978, IVF
has unde gone subs an ial ad ancemen s, esul ing in imp o ed success a es and o e ing hope o
hose s uggling wi h na u al concep ion [1].
In e ili y has eme ged as a majo public heal h conce n in indus ialized na ions [
2
]. This end is
closely linked o he ise o unheal hy li es yle beha io s cha ac e is ic o mode n socie y [
3
]. Fac o s
such as physical inac i i y, poo die a y habi s, and excess body weigh a e nega i ely associa ed wi h
ep oduc i e heal h, con ibu ing o a decline in e ili y po en ial [
4
]. In addi ion, en i onmen al
exposu es, o example endoc ine-dis up ing chemicals, ha e been shown o impai bo h emale and
male ep oduc i e unc ion [5].
The p ocedu e o IVF includes o a ian s imula ion, egg e ie al, e iliza ion in a labo a o y
en i onmen , emb yo cul u e, and subsequen emb yo ans e o he u e us [
6
]. Fe iliza ion occu s
h ough ei he s anda d IVF o In acy oplasmic Spe m Injec ion (ICSI). In s anda d IVF, eggs and
spe m a e combined in a cul u e dish, allowing e iliza ion o occu na u ally as i would in he
emale ep oduc i e ac . This me hod elies on he spe m’s abili y o pene a e he egg wi hou
medical in e en ion and is p ima ily used in cases whe e spe m quali y is su icien o na u al
e iliza ion. In con as , ICSI in ol es he di ec injec ion o a single spe m cell in o he cy oplasm
o a ma u e oocy e, allowing e iliza ion o occu e en in cases o se e e male- ac o in e ili y,
howe e , i is inc easingly used o all ypes o in e ili y, accoun ing o now 70-80% o ART cycles
globally [7].
A e e iliza ion, emb yos a e cul u ed in incuba o s ha main ain s able en i onmen al condi ions,
including empe a u e, gas composi ion, and pH le els, which a e essen ial o op imal emb yonic
de elopmen . Mode n ime-lapse incuba o s, such as he Emb yoScope (Vi oli e©, Sweden), o e
con inuous image acquisi ion wi hou emo ing emb yos om hei con olled en i onmen . This
unin e up ed cul u e combined wi h eal- ime moni o ing allows emb yologis s o e alua e bo h
mo phological cha ac e is ics and mo phokine ic pa ame e s - he iming o cell di isions and
o he de elopmen al miles ones. These ime-lapse ea u es ha e been associa ed wi h imp o ed
emb yo selec ion, as mo e equen obse a ions yield deepe insigh s in o de elopmen al compe ence,
implan a ion po en ial o ch omosomal no mali y [8].
9
Chap e 1. In oduc ion, Mo i a ion and Objec i es
A key decision in ART is whe he o ans e emb yos a an ea ly s age o allow u he in i o
de elopmen be o e selec ion. Ea ly-s age ans e occu s a he clea age s age, ypically on day
wo o h ee, when he emb yo comp ises ou o eigh cells. This app oach is o en p e e ed when
emb yo numbe s a e limi ed, as only abou hal o all emb yos each he blas ocys s age unde
ex ended cul u e condi ions. In con as , blas ocys -s age ans e , pe o med on day i e o six,
o e s imp o ed implan a ion po en ial due o enhanced emb yonic de elopmen and sel -selec ion.
Howe e , no all emb yos success ully de elop o he blas ocys s age, and in some cases, especially
among pa ien s wi h a limi ed numbe o emb yos, his may esul in no emb yos a ailable o
ans e [9].
The selec ion o emb yos, as well as he decision be ween ea ly-s age o blas ocys -s age ans e ,
elies hea ily on he expe ise o emb yologis s. These decisions a e based on he assessmen o
bo h mo phokine ic and mo phological cha ac e is ics o es ima e which emb yo has he highes
po en ial o implan a ion o success ul de elopmen o he blas ocys s age. Howe e , his e alua ion
p ocess is inhe en ly subjec i e and suscep ible o human bias. Va ia ions in expe ience, aining,
and in e p e a ion among clinicians can lead o inconsis encies in emb yo selec ion, ul ima ely
a ec ing implan a ion and p egnancy ou comes. In e ospec i e analyses, some A i ical In elligence
(AI)-based models ha e demons a ed p omising pe o mance, wi h epo ed accu acies exceeding
hose o manual e alua ions by emb yologis s [10].
In his hesis, he p edic i e capabili ies o a ious AI models o o ecas ing blas ocys o ma ion
om day 3 emb yos a e in es iga ed. The goal is o de elop a model capable o de e mining
whe he an emb yo will success ully each a good-quali y blas ocys s age, he eby suppo ing clinical
decisions on whe he o p oceed wi h ea ly-s age o ex ended emb yo cul u e and ans e . A da ase
comp ising 10,274 ime-lapse images o emb yos was assembled as he ounda ion o his s udy. An
e alua ion will be conduc ed o de e mine which ypes o da a and modeling app oaches o e he
highes p edic i e powe , using a ange o neu al ne wo ks and machine lea ning echniques applied
o di e en da a modali ies.
This wo k was ca ied ou du ing a mas e ’s hesis a he Ba celona Supe compu ing Cen e (BSC)
and Uni e si a Poli ècnica de Ca alunya (UPC), as pa o a collabo a i e p ojec be ween he BSC
and he Hospi al Clínic de Ba celona.
10
Chap e 2
S a e o he A
2.1 Mo phology and Mo phokine ics
S udies ha e shown ha bo h mo phological - he emb yo’s isual quali y a ixed ime poin s in
hou s pos insemina ion (HPI) - and mo phokine ic in o ma ion - he iming o de elopmen al e en s
such as he iming o cell di isions - can help de e mine he p obabili y o blas ocys o ma ion.
Wong e al. [
11
] epo ed ha emb yos exhibi ing as e ea ly clea age imes we e signi ican ly mo e
likely o de elop in o high-quali y blas ocys s. This suppo s he hypo hesis ha mo phokine ics
ca y p ognos ic alue. Howe e , he s udy was limi ed by a ela i ely small da ase .
In clinical p ac ice, emb yo assessmen elies hea ily on mo phology-based g ading sys ems. Ea ly-
s age e alua ion includes he zygo e sco e (Z-sco e) - which assesses he quali y o he zygo e sho ly
a e e iliza ion - and he day 3 emb yo mo phology sco e, bo h o which ha e been shown o
co ela e wi h emb yo iabili y a la e s ages [
12
]. This highligh s ha mo phology a di e en
ime poin s du ing emb yo de elopmen is a s ong indica o o blas ocys o ma ion. A la e
s ages, clinicians use o he well-es ablished mo phology-based sco ing sys ems such as he Ga dne
g ading sys em and he Veeck and Zanino ic g ading scheme o e alua e blas ocys quali y based on
expansion, inne cell mass (ICM), and ophec ode m (TE) s uc u e. While high-sco ing blas ocys s
ypically demons a e highe implan a ion po en ial, i is impo an o no e ha e en lowe -sco ing
emb yos can s ill esul in success ul p egnancies, howe e wi h a dec eased p obabili y [13].
Figu e 2.1:
Selec ed ames om a ime-lapse sequence o an emb yo de eloping o he blas ocys s age,
anno a ed wi h ime in HPI and co esponding mo phokine ic s ages. The ed ma ke indica es
he app oxima e p edic ion ime conside ed in his hesis, a which he likelihood o eaching a
good-quali y blas ocys should be assessed. Fo de ails on de elopmen al s age anno a ions, see
sec ion 3.1.
11
Chap e 2. S a e o he A
2.2 AI Models o Emb yo Assessmen
As men ioned in he in oduc ion, AI models ha e been applied o a ious asks ela ed o emb yo
assessmen . These models can suppo expe s by educing he need o ex ensi e manual anno a ion
and labou and may help mi iga e human bias and a iabili y in decision-making.
2.2.1 Emb yo Labelling, Sco ing and Selec ion
Di e en s udies a emp o p edic sco es o labels o emb yos. Fo example, [
14
] p edic ed
mo phokine ic anno a ions o indi idual ames om ime-lapse sequences wi h an accu acy o
84.79% wi h a da ase o 1,309 emb yos, while [
15
] coun ed blas ome es om 1 o 5 cells wi h a
modi ied U-Ne ained on 190 images, epo ing an a e age accu acy o 88.2%. [
16
] p edic ed
emb yo quali y a day 3 in ou ca ego ies, ou pe o ming emb yologis s wi h an ensemble lea ning
model based on a ious Con olu ional Neu al Ne wo k (CNN) ne wo ks, ained on 3,601 mic oscopic
images. The model o code o he p e iously men ioned s udies a e no published. The STORK
model [
17
] p edic s he inal Veeck and Zanino ic blas ocys quali y g ading om a blas ocys image
using a p e ained Incep ion-V1 CNN, ained wi h a da ase o 10,148 human emb yos. The code
is a ailable and he model accessible hough a web in e ace.
One o he mos common applica ions o AI in ep oduc i e medicine is emb yo selec ion, whe e
models a e ained o p edic he iabili y o emb yos a he blas ocys s age (day 5) o ans e .
The classi ica ion ask is ypically based on implan a ion success o li e bi h ou comes. Ve Milyea
e al. [
18
] de eloped he Li e Whispe e AI sys em using ac ual p egnancy ou comes as g ound u h
da a. The sys em is comme cially a ailable as a decision-suppo ool o emb yo selec ion and uses
s a ic wo-dimensional op ical mic oscope images o day 5 blas ocys s as inpu . In he con ex o he
o iginal s udy and subsequen e ospec i e analyses, he model ou pe o med emb yologis s [
19
].
I was ained on 8,886 emb yos collec ed om 11 IVF clinics. One o he mos complex models,
iDASco e 2.0, was de eloped by Lassen e al. [
20
]. I u ilizes sepa a e Two-S eam In la ed 3D
Con Ne (I3D) ideo models [
21
] o day 2/3 and day 5+ emb yos, p ocessing 64 o 128 ames
espec i ely om ime-lapse sequences. The model p edic s implan a ion po en ial wi hou equi ing
manual anno a ions. T ained on a da ase o 181,428 emb yos om 22 IVF clinics wo ldwide, i
ep esen s he la ges da ase used o emb yo model aining o da e. Inclusion o disca ded emb yos
in he aining da a enhanced he model’s abili y o di e en ia e be ween emb yos sui able o
ans e and hose o be disca ded, a oiding bias owa ds only high-quali y emb yos wi h known
implan a ion ou comes. iDASco e 2.0 is a ailable as an op ional so wa e module o Emb yoScope
ime-lapse incuba o s.
Al hough he p edic ion goals o hese s udies di e om ha o his hesis, hey o e aluable
e e ence poin s by demons a ing how deep lea ning can ex ac meaning ul isual and empo al
pa e ns om emb yo image y. These app oaches highligh s a egies o model design and da a
handling ha a e ele an o he p edic i e ask add essed in his wo k.
2.2.2 Blas ocys P edic ion
The p edic ion ask o his wo k - he o ma ion p obabili y o good-quali y blas ocys s om day 3
emb yos - was p e iously add essed by [
22
]. The ensemble model, called STEM, eached 0.82 A ea
Unde Cu e (AUC), wi h a balanced accu acy (see o mula 5) o 76.1%. The model consis s o a
mo phological s eam model, a DenseNe ea u e ex ac o using i e ixed ime poin s s a ing om
PNF, and a empo al s eam model, coun ing cells in each ame and eeding his in o ma ion in o
an Long Sho -Te m Memo y (LSTM) ne wo k. The da ase consis ed o 10,432 emb yo ime-lapse
12
Chap e 2. S a e o he A
ideos. They no ed ha clea age ime di e s signi ican ly be ween ICSI and con en ional IVF
emb yos, as p e iously men ioned in he in oduc ion. The e o e, hey employed a
PNF
ecogni ion
ne wo k o align he ime-lapse sequences o his e e ence poin .
An in e es ing s udy by [
23
] in oduced a no el app oach o assessing blas ocys o ma ion based on
cy oplasmic pa icle mo emen du ing he i s 44 hou s o emb yo de elopmen . Using ime-lapse
imaging combined wi h Pa icle Image Velocime y, he s udy ex ac ed emb yo mo emen ec o s
by analyzing pixel-le el changes be ween consecu i e ames. These mo ion-de i ed ea u es we e
used as inpu o a ious machine lea ning models, including LSTM neu al ne wo ks and k-Nea es
Neighbo s, o p edic he likelihood o blas ocys de elopmen . I is pa icula ly no able since no
in o ma ion abou he emb yo’s ac ual appea ance o mo phokine ic ime poin s was used. The inpu
da a consis ed solely o ea u es de i ed om cy oplasmic mo emen , such as summed pixel-wise
mo ion ec o s, wi hou inco po a ing appea ance-based o mo phokine ic anno a ions. Howe e ,
he da ase was small (230 emb yos) and based on a highly selec i e scena io - sibling emb yos whe e
only one de eloped o blas ocys - limi ing i s eal-wo ld applicabili y.
Ano he s udy [
24
] epo ed a balanced accu acy o 66% o p edic ing i an emb yo de elops in o a
blas ocys , using a Xcep ion CNN. The inpu is a single image a 70 HPI sampled om a da ase o
3,469 emb yo ime lapse images.
The s a e-o - he-a model was p oposed by [
25
] wi h hei Adap i e Key F ame Selec ion (AdaKFS)
app oach, which combines an LSTM and a policy ne wo k. The s udy used a da ase o 3,300 human
emb yo ime-lapse ideos, wi h each emb yo labeled o blas ocys o ma ion ou come based on he
Ga dne sco ing sys em. F om each ideo, 32 ames we e uni o mly sampled and passed h ough a
ResNe -50 backbone o ex ac mo phological ea u es. These ea u es we e combined wi h encoded
kine ic pa ame e s and inpu in o an LSTM model. A each ime s ep, a policy ne wo k decided
whe he o skip o e ain he cu en ame’s hidden s a e o p edic ion. The selec ed hidden
s a es (on a e age a ound six pe ideo) we e hen passed o a p edic ion ne wo k o es ima e he
p obabili y o blas ocys o ma ion. The me hod was e alua ed agains he 8-laye CNN app oach by
[
26
] and he STEM ne wo k by [
22
] on hei da ase , ou pe o ming bo h wi h a balanced accu acy
o 69.43%. A baseline model using only he LSTM wi hou ame selec ion achie ed a lowe accu acy
o 60.77%.
The mos ecen publica ion by [
27
] was ained on he public da ase in oduced by [
28
], which
con ains 704 ime-lapse ideos. O hese, 522 we e labeled as alid blas ocys s — an a guably
ques ionable numbe , as a close analysis o he da ase e eals ha only 490 emb yos eached he
B
s age, 388 eached
EB
, and jus 5 we e anno a ed wi h
HB
. Despi e his, he au ho s epo a
no ably high accu acy o 93% using a ine- uned ResNe -50 on ame-wise mo phokine ic anno a ions
combined wi h a GRU-based sequence model. Howe e , his esul is ques ionable due o he small
da ase size and inconsis ency wi h pe o mance me ics epo ed in o he s udies. This public
a ailable da ase was no selec ed o his mas e ’s hesis because i does no accoun o emb yo
iabili y. I is in ended solely o ame-wise phase p edic ion, as no ed by [
28
], and is he e o e
unsui able o ou classi ica ion goal o iden i ying high-quali y blas ocys s.
Un o una ely, none o he men ioned models aligning wi h ou p edic ion ask ha e been made
publicly a ailable. Only he code o he STEM ne wo k p oposed by [
22
] has been eleased. As a
esul , di ec compa ison is challenging, pa icula ly due o di e ences in da ase s and he lack o
comple e implemen a ion de ails.
13

Chap e 3
Da ase
Th ough a o mal ag eemen , he BSC was g an ed access o a da abase o emb yo ime-lapse images
p o ided by he Hospi al Clínic de Ba celona, which includes a secu e p o ocol o da a ans e ence
and s o age. This chap e desc ibes he p ocess o assembling he da ase and p epa ing he da a o
model aining in he subsequen chap e s. The s a is ics o he da ase can be ound in Appendix B.
3.1 Me ada a
Each emb yo in he hospi al’s da abase is s o ed wi h a unique ID, wi h eco ds da ing back o
2014 and con inuing o he p esen (No embe 2024). In o al, 53,123 emb yos a e egis e ed in he
da abase. O hese, 48,714 emb yos ha e me ada a a ailable, which a e conside ed o u he
analysis.
Due o da a p i acy egula ions, pa ien in o ma ion was excluded. Only he unique Emb yoID
was e ained. As a esul , he da ase con ains no pe sonal de ails such as pa ien age o e hnici y -
his in o ma ion was no a ailable in he o iginal da a. Addi ionally, i is no possible o de e mine
whe he mul iple emb yos o igina ed om he same pa ien .
The me ada a includes he ollowing in o ma ion:
•
P onuclea (PN) Numbe : This alue indica es he numbe o p onuclei in he emb yo a e
insemina ion. A PN o 2 is conside ed no mal, while o he alues a e classi ied as abno mal
and may sugges poo quali y.
•
Ins umen Numbe : The hospi al uses i e di e en ime-lapse incuba o s, o wo ypes:
Emb yoScope+ and Emb yoScope-D. These sys ems di e in image esolu ion - Emb yoScope+
p oduces 800
×
800 pixel images, while Emb yoScope-D p oduces 500
×
500 pixel images. The
c opping o he well also di e s: in he Emb yoScope+, he well appea s la ge . Op ical bias
may also be in oduced due o di e ences in he imaging sys ems, as well as en i onmen al
a ia ion esul ing om a chi ec u al di e ences be ween he incuba o s.
•
Emb yo Fa e: This pa ame e deno es he inal s a us o he emb yo. Possible ou comes
include:
–A oid: The emb yo was disca ded.
–F eeze: The emb yo was deemed iable and ozen o u u e use.
15
Chap e 3. Da ase
–Unknown: The a e was ei he no anno a ed o labeled as ’unknown’.
–T ans e : The emb yo was conside ed high quali y and was implan ed.
–Undecided: I was unclea whe he he emb yo was o su icien quali y.
•
Mo phokine ic Anno a ions: These anno a ions ma k he ime poin s o key e en s in
emb yo de elopmen . Below is a summa y o all mo phokine ic ime poin s, whe e each o
hem excep Dead mus appea in ch onological o de .
– PB2: Time o he second pola body ex usion.
– PNa: Time when p onuclei appea .
– PN : Time when p onuclei ade.
– n
(
2, 3, 4, . . . , 9
): Rep esen s he ime poin s when he emb yo unde goes successi e
cell di isions, whe e
2
indica es he i s di ision (2-cell s age),
3
indica es he 3-cell
s age, and so on up o 9.
– SC
: S a o compac ion, whe e indi idual cells begin o me ge in o a compac s uc u e.
– M: Mo ula s age, whe e he emb yo consis s o a igh ly packed mass o cells.
– SB: S a o blas ula ion, whe e he i s signs o blas ocys o ma ion appea .
– B: Blas ocys o ma ion, indica ing he de elopmen o a luid- illed ca i y.
– EB
: Expanded blas ocys s age, whe e he blas ocys enla ges in p epa a ion o implan-
a ion.
– HB: Ha ching blas ocys , when he blas ocys begins o b eak ou o i s zona pellucida.
– Dead
: Time poin when doc o s de e mine ha he emb yo has ceased de elopmen . This
e en is no necessa ily in ch onological o de , as i can be eco ded a any ime.
I is no ed ha a s anda dized emb yo quali y assessmen like he Ga dene Sco e is no p esen in
he me ada a. O he s udies used his me ic as he main ac o o label he emb yos as ei he a
blas ocys wi h good quali y o bad quali y, which makes he accu acy and labels compa able o
o he s udies.
Fu he mo e, wo c i ical pieces o in o ma ion a e missing o he emb yos: he pa ien ’s age and
he e iliza ion me hod. The o me is a key de e minan o blas ocys o ma ion a es, wi h he
pe cen age o emb yos pe pa ien eaching he blas ocys s age dec easing linea ly as ma e nal age
inc eases [
29
,
30
]. In con as , he e iliza ion me hod has no signi ican impac on he blas ocys
o ma ion a e o he quali y o he esul ing blas ocys [
31
]. Howe e , IVF-de i ed emb yos each
he blas ocys s age signi ican ly as e han hose e ilized ia ICSI [
32
]. As discussed in chap e
2, he a e o emb yonic de elopmen se es as an impo an indica o o success ul blas ocys
o ma ion. Omi ing his in o ma ion om he model p e en s i om dis inguishing whe he an
emb yo’s de elopmen is abno mally slow o as o whe he he obse ed di e ences a e simply due
o a ia ions in he e iliza ion me hod.
16
Chap e 3. Da ase
3.2 Emb yo Sample Selec ion
The selec ion o emb yos o his s udy was cons ained by bo h echnical and biological ac o s.
Each emb yo consis s o app oxima ely 500 ames, cap u ed ac oss mul iple ocal planes (ei he 7
o 11, symme ically dis ibu ed a ound he cen e ocal plane), wi h spacing o ei he 15 µm o
25 µm depending on he Emb yoScope imaging sys em. Due o secu i y p o ocols, downloading a
single emb yo ook app oxima ely one minu e, making he e ie al o he ull da ase in easible
wi hin he a ailable ime. Fu he mo e, a e he ini ial selec ion desc ibed in his chap e , echnical
es ic ions p e en ed u he access o he hospi al da abase, limi ing he da ase o he emb yos
al eady ob ained.
To ensu e ha only ele an emb yos we e included, all emb yos ha did no each he 72-hou
ma k acco ding o he anno a ions we e excluded, as his was he equi ed p edic ion ime poin .
Figu e 3.1:
Mos ecen ime poin o anno a ion in hou s o all 48,714 emb yos. A black line is added a
he p edic ion ime o 72 hou s. All emb yos le o he line we e disca ded.
Since images we e no ye downloaded a his s age, he de elopmen al ime o each emb yo was
es ima ed based on he imes amp o i s mos ecen anno a ion. I no u he anno a ions we e
eco ded, i was assumed ha he emb yo did no p og ess beyond ha poin . A e applying his
il e , 17,140 emb yos emained, ep esen ing 35.1% o he o iginal 48,714 emb yos wi h alid
me ada a. These emb yos we e downloaded om he Hospi al and we e a ailable o he da ase .
This selec ion p ocess in oduces a po en ial bias in he da ase . The absence o anno a ions beyond
72 hou s could be due o se e al easons:
•
P e-72h A es : Some emb yos ailed o de elop be o e eaching 72 hou s and we e conside ed
non- iable ea ly on. Since hese cases do no equi e model-based decisions a day 3, hei
exclusion was alid.
•
T ans e /C yop ese a ion: Some emb yos we e ans e ed o a pa ien o c yop ese ed
be o e eaching he blas ocys s age a day 3. While hese emb yos we e likely o good quali y
a 72 hou s, hei inal blas ocys de elopmen al ou come emains unknown and hey should
be excluded.
•
Disca ded a 72h: Some emb yos may ha e been deemed non- iable jus a 72 hou s, leading
o he absence o u he anno a ions. I imaging da a a 72 hou s exis ed o hese emb yos,
hey could ha e been labeled as False cases. Howe e , since hey we e no included, he model
was ne e ained on emb yos ha we e explici ly disca ded a his s age.
This esul s in a da ase whe e he model only lea ns om emb yos ha appea ed iable a 72
17
Chap e 4. Me hods
RadImageNe p o ides a po en ially mo e e ec i e s a ing poin o ea u e ex ac ion han na u al
image da ase s.
All models in his s udy a e e alua ed unde h ee ini ializa ion se ings: wi h ImageNe -p e ained
weigh s, wi h RadImageNe -p e ained weigh s, and wi h andom ini ializa ion. This allows o a
sys ema ic assessmen o how di e en p e aining s a egies in luence pe o mance on he gi en
ask.
Image Assembly
The da ase comp ises ime-lapse sequences o g ayscale mic oscopic images cap u ed a a ious
ocal planes. Using a single image as inpu o he CNN ne wo ks discussed in sec ion 4.2.1 is no
di ec ly possible, since mos con olu ional neu al ne wo ks o image classi ica ion a e designed o
3-channel RGB inpu s. To enable compa ibili y wi h such a chi ec u es, he single-channel g ayscale
images mus be expanded o h ee channels. Se e al s a egies a e e alua ed o his pu pose, each
o e ing di e en o ms o spa ial o empo al con ex :
1.
S a ic g ayscale: The cen al ocal plane is duplica ed ac oss all h ee channels. This
con igu a ion does no add any addi ional spa ial o empo al con ex .
2.
Focal planes: Fo each ame, he cen al ocal plane is combined wi h one plane abo e and
one below, o ming a h ee-channel inpu ha cap u es axial dep h in o ma ion. The in e -
plane spacing is ei he
±
25 µm o
±
15 µm, depending on he imaging sys em (Emb yoScope+
o Emb yoScope D, espec i ely).
3.
Tempo al sequence: The cu en ame a p edic ion ime is combined wi h he wo
p eceding ames om he ime-lapse sequence. This con igu a ion in oduces sho - e m
empo al in o ma ion and cap u es sub le emb yonic mo emen s occu ing o e a ime span o
app oxima ely 15 minu es o one hou , depending on he image acquisi ion in e al.
G ad-CAM Ac i a ion Maps
To imp o e in e p e abili y o he CNNs used in his s udy, G adien -weigh ed Class Ac i a ion
Mapping (G ad-CAM) is employed. G ad-CAM is a widely used isualiza ion echnique ha
highligh s he egions o an inpu image ha con ibu e mos o a model’s decision [
52
]. I does
so by using he g adien s o he a ge class lowing in o he inal con olu ional laye o p oduce a
coa se localiza ion map o impo an egions.
Fo he ained models, G ad-CAM is applied o a andom ba ch o alida ion images in o de o
quali a i ely assess which image egions he model ocuses on when making a classi ica ion decision.
This is pa icula ly aluable in biomedical imaging, whe e model decisions should ideally co ela e
wi h biologically meaning ul s uc u es. The esul ing hea maps a e o e laid on he o iginal inpu
images, enabling isual inspec ion o whe he he ne wo k’s ocus aligns wi h expec ed mo phological
ea u es.
In his s udy, G ad-CAM is applied using he inal con olu ional laye o each model, o he
E icien Ne B0-Model i is namely he
ea u es[7]
laye , as his egion o he model e ains spa ial
in o ma ion while s ill inco po a ing high-le el seman ic ea u es. All isualiza ions a e gene a ed
pos hoc, wi hou modi ying he ne wo k a chi ec u e o a ec ing model pe o mance. This me hod
p o ides an in ui i e and model-agnos ic app oach o explainabili y, complemen ing e alua ion
me ics wi h quali a i e insigh in o he model’s decision-making p ocess.
24

Chap e 4. Me hods
Augmen a ion Techniques
Du ing he cou se o his wo k, se e al issues wi hin he da ase we e iden i ied:
•Random obscu a ion esul ing in e y low b igh ness in ce ain ames
•Emb yo wells no cen e ed o pa ially c opped
•Ou -o - ocus ames
•JPEG comp ession a i ac s
Al hough such cases a e ela i ely a e, hey pose a isk o o e i ing, as he model may easily
memo ize hese a ypical samples due o hei dis inc i e isual ea u es. Gi en he la ge size o he
da ase and he limi ed ime a ailable, manual inspec ion was no easible. Ins ead, augmen a ion
echniques a e in oduced o mimic he inconsis encies ound in he ou lie images. The goal was o
mi iga e o e i ing on a ypical samples and inc ease he gene aliza ion o he model.
Figu e 4.2:
G ad-CAM O e lay o he model p edic ion on a selec ion o samples (column-wise). The image
on he le includes blu on he igh side o he ame and he model is ocusing on he blu
ou side o he well. The wo ames o he igh show no mal G ad-CAMs ocusing co ec ly on
he emb yo, how i was obse ed o mos examples. On he igh , wo ames wi h obscu a ion
a e shown, whe e as well he model does no ocus on he emb yo e en ho he con ou s a e
sligh ly isible.
Ano he issue is he p esence o non-uni o mi y ou side o he emb yo well, which may con ain
dis inc ea u es and in oduces bias. G ad-CAM isualiza ions e ealed ha he model occasionally
ocused on hese ex e nal egions a he han he emb yo i sel . To add ess his issue, an algo i hm
was de eloped o black ou he a ea ou side he well and cen e he well wi hin he image. The
p ocedu e is desc ibed in Appendix C. Howe e , due o he da ase ’s size and a iabili y, a numbe o
samples lack a de ec ed well ci cle and emain unp ocessed. Applying c opping o all ames in such
cases led o a dec ease in pe o mance. The e o e, c opping was applied wi h a p obabili y o 50%.
This p obabilis ic app oach imp o ed pe o mance while p o iding wo key ad an ages. Fi s , images
wi hou a de ec ed well a e no pa icula ly s anding ou , p e en ing he model om o e i ing
on hese ew unique examples. By no c opping e e y image, he model lea ns o emain obus o
bo h c opped and unc opped inpu s. Second, he in e ence da ase does no equi e c opping, as he
25
Chap e 4. Me hods
model pe o ms well on unp ocessed inpu s. This lexibili y simpli ies deploymen and allows he
model o handle a b oade ange o inpu o ma s.
4.2.2 Emb yo Fea u e Model
Figu e 4.3:
O e iew o he Emb yo Fea u e Model, combining he anno a ions om he medical expe s
wi h image inpu . The ea u e ec o has a leng h o 24, as desc ibed in Sec ion 3.4. The image
inpu consis s o one ame sampled 0 o 40 ames be o e p edic ion ime a 72 HPI and he
wo p eceding ames. Yellow blocks indica e p e ained componen s.
Figu e 4.4:
Scaled Do P oduc a en ion block om igu e 4.3 in de ail. Que y (Q), Key (K) and Value
(V) a e linea p ojec ions om he inpu s A o B.
Nex , i was e alua ed whe he he anno a ion ea u es (see Sec ion 4.1) could enhance model
pe o mance when combined wi h an image inpu . While he anno a ions cap u e he mo phokine ics,
hey do no con ain mo phological in o ma ion. Con e sely, a single image p ima ily e lec s
mo phology, al hough i may implici ly con ey de elopmen al speed based on he emb yo’s appea ance
a p edic ion ime.
26
Chap e 4. Me hods
To in eg a e bo h in o ma ion sou ces, he anno a ion ea u e ec o was p o ided as a seconda y
inpu o he model, p ocessed by a dedica ed subne wo k alongside a single image inpu handled by
he backbone CNN, as i was desc ibed in he p e ious sec ion. The subne wo k consis ed o a single
ully-connec ed (FC) laye wi h 1024 dimensions. I was p e ained ollowing he same s a egy
ou lined in Sec ion 4.1, bu implemen ed as a neu al ne wo k wi h a single ou pu node a he han
a adi ional ML model.
The key challenge lies in e ec i ely using he ou pu s o bo h ne wo ks. Se e al usion s a egies
we e e alua ed:
•
Conca ena ion: The ep esen a ions om he image and ea u e modali ies a e conca ena ed
and passed o a classi ie . Addi ionally, a Modali yD opou a ian was explo ed, whe e one
modali y is andomly masked du ing aining. This encou ages he model o lea n obus
ep esen a ions om bo h modali ies and educes o e - eliance on a single inpu sou ce.
•
Mix u e o Expe s (MoE): A ga ing ne wo k p ocesses he conca ena ed modali y inpu s
and lea ns o assign weigh s o each modali y-speci ic inpu . The inal ep esen a ion is
a weigh ed sum o he ou pu s om he wo ne wo ks, allowing he model o dynamically
p io i ize in o ma ion om di e en modali ies depending on he inpu [53].
•
A en ion-Based Fusion: The wo ou pu s a e passed h ough an a en ion block [
42
]. Each
modali y (A and B, wi h 1024-dimensional inpu s) is linea ly p ojec ed o a que y, key, and
alue space wi h 512 hidden dimensions. C oss-a en ion is compu ed by pe o ming scaled
do -p oduc a en ion om Que y A o Key B and ice e sa, each ollowed by a So max
ope a ion. These a en ion maps a e hen applied o he co esponding alue p ojec ions
(Value B and Value A). A skip connec ion adds he o iginal modali y inpu s back o hei
espec i e a en ion ou pu s. Howe e , he inpu s a e also p ojec ed o 512 dimensions in o de
o ma ch he dimensions. The esul ing ep esen a ions a e laye -no malized and conca ena ed.
The scheme can be seen in igu e 4.3 and 4.4.
I is addi ionally explo ed i a sha ed FC laye (as seen in igu e 4.3) imp o es he pe o mance o he
model. This laye enables join ep esen a ion lea ning by p ojec ing bo h modali ies - image and
anno a ion ea u es - in o a common la en space. Such sha ed laye s ha e been shown o imp o e
mul imodal alignmen and educe modali y-speci ic o e i ing in ela ed wo k [54].
4.2.3 Emb yo Video Model
Figu e 4.5:
O e iew o he comple e Emb yo Video Model. The inpu consis s o 20 ames pe sample,
each p ocessed in o a 1280-dimensional ea u e ec o by he E icien Ne B0 backbone. The
inal ou pu is a single classi ica ion p edic ion.
27
Chap e 4. Me hods
The empo al e olu ion o an emb yo con ains signi ican ly mo e in o ma ion han a single image
can p o ide. As shown in Sec ion 5.2.2, combining mo phokine ic anno a ions wi h image inpu
yielded he bes pe o mance so a . Howe e , hese anno a ions mus be manually ex ac ed by
ained p o essionals, in oducing subjec i i y and manual labou . The objec i e is o de elop a
model capable o making p edic ions au oma ically, wi hou elying on manually cu a ed ea u es.
To implici ly cap u e mo phokine ic cha ac e is ics, sequen ial image da a om he ime-lapse sys em
is le e aged. A LSTM ne wo k is employed in conjunc ion wi h a backbone ea u e ex ac o o
p ocess he empo al dynamics o emb yo de elopmen . LSTMs a e a class o ecu en neu al
ne wo ks designed o model sequen ial da a by main aining an in e nal memo y o p io inpu s.
Unlike models ha ea each inpu independen ly, LSTMs can cap u e empo al dependencies
by upda ing a hidden s a e
h
and a cell s a e
c
a each ime s ep
. This is achie ed h ough
ga ed mechanisms - namely, he inpu , o ge , s a e candida e and ou pu ga es - which egula e he
low o in o ma ion and enable he ne wo k o e ain o disca d speci ic ea u es o e ime. These
p ope ies help mi iga e challenges such as he anishing g adien p oblem, allowing he model
o lea n long- ange empo al pa e ns e ec i ely [
55
]. Consequen ly, LSTMs a e well-sui ed o
modeling emb yo de elopmen , whe e he iming and o de o mo phokine ic e en s a e essen ial o
accu a e p edic ion.
Figu e 4.6:
De ailed s uc u e o he modi ied LSTM cell o he Video Model. The di e en ga es a e
indica ed by do ed ou lines. The symbol
σ
deno es he sigmoid ac i a ion unc ion, while
anh
ep esen s he hype bolic angen . b e e s o bias ec o s, W o inpu weigh ma ices, and R
o ecu en weigh ma ices. The ec o s h
−1
and c
−1
ep esen he hidden and cell s a es
om he p e ious ime s ep, espec i ely, and x
is he inpu ea u e ec o a he cu en ime
s ep .
The bes -pe o ming E icien Ne B0 model om Sec ion 4.2.1 is used as he ea u e ex ac o ,
p oducing a 1,280-dimensional ea u e ec o o each ame in he ime-lapse sequence. A o al o 20
ames a e sampled a uni o m in e als and passed sequen ially o he LSTM ne wo k, wi h he inal
ame co esponding o he p edic ion ime a 72 HPI. Due o compu a ional and un ime cons ain s,
28
Chap e 4. Me hods
inco po a ing mo e ames was no easible. To in es iga e he e ec o empo al esolu ion, di e en
ame spacing s a egies we e es ed. Fo classi ica ion, only he las hidden s a e
h
o he LSTM
- co esponding o he inal ime s ep - is used as inpu o he classi ica ion head. In he inal
a chi ec u e, he backbone, LSTM, and classi ie head we e ained join ly in an end- o-end manne ,
meaning ha he backbone was no simply ac ing as a ozen ea u e ex ac o . This signi ican ly
inc eased he compu a ional load; howe e , i allowed he backbone o adap o emb yo appea ances
a p e iously unseen ime poin s, as he p e ained backbone had o iginally only been ained wi h
images cap u ed nea he p edic ion ime.
Ini ial expe imen s showed signi ican o e i ing, due o inc eased model complexi y and educed
da a augmen a ion ela i e o he in o ma ion densi y in longe ime sequences. A e ex ensi e
expe imen s, imp o ed gene aliza ion and s able aining dynamics we e achie ed h ough he
ollowing egula iza ion echniques:
•Single-laye LSTM wi h 512 hidden uni s
•D opou a e o 0.2 applied o he LSTM hidden s a e h
•Laye no maliza ion applied o bo h he LSTM cell ou pu c and hidden s a e h
•Weigh decay o 1×10−5applied o he inpu - o-hidden weigh s W , Wi, Wo, Wg
•A educed ully connec ed laye in he classi ie head wi h 256 hidden uni s
•D opou o 0.2 applied o he ully connec ed laye (as al eady applied be o e)
29

Chap e 5
Resul s
In his chap e , a se ies o expe imen s a e conduc ed o compa e di e en models, da a augmen a ion
s a egies, and hype pa ame e con igu a ions. All esul s will be join ly analyzed in chap e 7 and
a e summa ized in able 7.1.
Fo o e all model selec ion, balanced accu acy is used as he p ima y e alua ion me ic. This choice
e lec s he ac ha he e is no equi emen o a o ei he he posi i e o nega i e class - he
objec i e is o op imize gene al pe o mance ac oss bo h. Balanced accu acy is de ined as he a e age
o sensi i i y ( ue posi i e a e) and speci ici y ( ue nega i e a e), and is compu ed as ollows:
Balanced Accu acy =1
2T P
T P +F N +T N
T N +F P (5.1)
5.1 Machine Lea ning Model
Expe imen s we e conduc ed using Py hon 3.10.15 on an Apple M1 Mac. The
RandomFo es Classi ie
om sciki -lea n 1.6.1 [
56
] was ained using de aul se ings. As inpu ,
he p ep ocessed ea u e ec o desc ibed in Sec ion 3.4 is used. Resul s a e shown in able 5.1.
Me ic Valida ion Se Tes Se
Accu acy 0.7203 0.7601
Sensi i i y 0.7695 0.8000
Speci ici y 0.6747 0.7230
Balanced Accu acy 0.7221 0.7615
Table 5.1:
Pe o mance me ics o he machine lea ning model (
RandomFo es Classi ie
) on he es and
alida ion se s.
5.2 Deep Lea ning Models
All deep lea ning models we e implemen ed using PyTo ch Ligh ning 2.4.0 [
57
], a high-le el w appe
o PyTo ch 2.5.0 (p e- elease, NVIDIA build n 24.10) [
58
]. T aining was conduc ed on he
31
Chap e 5. Resul s
Ma eNos um5 supe compu e , equipped wi h high-pe o mance compu ing (HPC) clus e s wi h
ou NVIDIA H100 GPUs (each wi h 64GB VRAM), using Py hon 3.10.12 and CUDA compila ion
ools elease 12.6 (V12.6.77).
Dis ibu ed aining was pe o med using Ligh ning’s Dis ibu ed Da a Pa allel (DDP) backend, wi h
mixed-p ecision aining enabled ia he
16-mixed
p ecision se ing. All ba ch no maliza ion laye s
we e con e ed o synch onized ba ch no maliza ion o ensu e consis en s a is ics ac oss de ices.
T aining was conduc ed o a maximum o 100 epochs, wi h ea ly s opping applied using a pa ience
o 30 epochs moni o ing he balanced alida ion accu acy. Fo each un, he model wi h he highes
balanced accu acy on he alida ion se was selec ed as he inal model. Final alida ion and es
p edic ions we e pe o med on a single GPU, and all epo ed me ics in ables ep esen he mean
o e mul iple aining uns wi h di e en andom seeds.
5.2.1 Single Image Model Resul s
A single ame pe emb yo is used as inpu o classi ica ion. A andom ame is selec ed om wi hin
a window o 40 ames p io o he ame a 72 HPI. This window can span up o 800 minu es,
depending on he image acquisi ion equency. Ini ial expe imen s showed op imal pe o mance
using his con igu a ion.
The ollowing basic ans o ma ion and augmen a ion pipeline is hen applied:
•The image is con e ed o a loa 32 enso wi h h ee channels.
•
Colo ji e ing is applied using o ch ision’s
ans o ms. 2.Colo Ji e
wi h b igh ness se
o 0.2 and con as se o 0.3.
•The image is esized o 512 ×512 pixels.
•Random ho izon al lipping is applied wi h a p obabili y o 0.5.
•A andom ixed o a ion o 0°, 90°, 180°, o 270°is applied.
•
The image is no malized using ImageNe s a is ics wi h mean = [0.485, 0.456, 0.406] and
s anda d de ia ion = [0.229, 0.224, 0.225].
All models a e implemen ed ia he
o ch ision.models
package ( e sion 0.20.0a0). Fo each
a chi ec u e, he inal classi ica ion laye is eplaced by an iden i y laye . A cus om classi ica ion
head is appended, consis ing o a ully connec ed laye wi h a hidden size o 1024, ollowed by a
ReLU ac i a ion and a d opou laye wi h a d opou a e o 0.2. A inal linea laye ou pu s a single
logi .
The bina y classi ica ion objec i e uses o ch’s
BCEWi hLogi sLoss
, which p o ides g ea e nume ical
s abili y han applying a sigmoid ac i a ion ollowed by BCELoss.
Addi ionally, label-smoo hing egula iza ion is employed by adding andom noise in he ange [0.0,
0.1] o he ha d labels (0 o 1), ollowing he me hod desc ibed in Sec ion 7 o [
46
]. This p e en s
he model om becoming o e ly con iden and imp o es gene aliza ion.
The AdamW op imize is used, wi h sepa a e lea ning a es and weigh decay pa ame e s o he
backbone and classi ica ion head. Weigh decay is only applied o he weigh s o con olu ional and
linea laye s. A wa m-up phase o 5 epochs is applied, ollowed by a cosine annealing lea ning a e
schedule , which g adually educes he lea ning a e o a common minimum by epoch 100.
32
Chap e 5. Resul s
The ollowing hype pa ame e s we e ound o p oduce good esul s ac oss he e alua ed models:
•Lea ning a e (backbone): 1×10−4
•Lea ning a e (classi ica ion head): 1×10−3
•Weigh decay (backbone): 1×10−6
•Weigh decay (classi ica ion head): 1×10−3
•Minimum lea ning a e: 1×10−5
•Ba ch size: 32
Backbone selec ion
Fo ini ial selec ion, uns whe e pe o med wi h ocal-plane images ( able 5.2). Howe e , as seen
in able 5.3, he Tempo al Sequence pe o med be e o e all. Only E icien Ne B0 p e ained on
ImageNe will conside ed as a s a ing poin o u he expe imen s.
Model None ImageNe RadImageNe
ResNe 50 59.09% 68.88% 63.74%
E icien Ne B0 59.33% 70.03% -
Con NeX -Base 53.09% 68.68% -
SwinV2-B 53.03% 68.27% -
Incep ionV3 64.30% 68.30% 61.49%
DenseNe 121 59.80% 69.10% 61.34%
VGG16 55.00% 68.03% -
Table 5.2:
Compa ison o model pe o mance (balanced accu acy on alida ion se ) ac oss di e en p e ained
weigh s, ei he andom weigh s, ImageNe weigh s o RadImageNe weigh s (whe e a ailable).
E icien Ne B0 pe o med he bes , ini ialized wi h ImageNe weigh s. Me ics ep esen he
mean o i e aining uns and images a e assembled using h ee ocal planes.
Inpu Type Balanced Accu acy
G ayscale (Duplica ed) 69.31%
Focal Planes 70.03%
Tempo al Sequence 71.41%
Table 5.3:
Pe o mance o E icien Ne B0 (ImageNe p e ained) using di e en image inpu con igu a ions.
Tempo al Sequence (p edic ion ame plus he wo p e ious ames) pe o med he bes o e all.
Me ics a e he mean o i e uns on he alida ion se .
Augmen a ion Techniques
Due o he i egula i ies in he da ase desc ibed in sec ion 4.2.1, i e new augmen a ion echniques
we e inco po a ed in o he basic ans o ma ion pipeline in oduced in Sec ion 5.2.1:
•P obabilis ic c opping o he emb yo well (50% chance), as desc ibed in Appendix C
33
Chap e 7. Discussion and Fu u e Wo k
Figu e 7.1:
G ad-CAM o e lays o wo ames om an LSTM inpu sequence. The le image co esponds
o he 4-cell s age and he igh o he 2-cell s age. The highligh ed egions align wi h indi idual
blas ome es, sugges ing he model is implici ly es ima ing cell coun om mo phology.
7.2 Using he Anno a ions as an Addi ional Lea ning Task
Al hough he Emb yo Video Model pe o med well, i did no su pass he Machine Lea ning Model
ained solely on manual anno a ions. This sugges s ha he Video Model alone does no ully
ex ac all mo phokine ic in o ma ion a ailable in he da ase . Tha would imply ha clinical
anno a ions - while subjec o human a iabili y - s ill ca y s ong p edic i e alue.
In e es ingly, despi e no being explici ly ained o iden i y cell s ages, he G ad-CAM esul s
indica e ha he model pa ially lea ns his ask (Figu e 7.1). Two po en ial imp o emen s o u u e
wo k would be hinkable o inco po a e he s ong in o ma i e alue o he manual anno a ions o
he da ase , while aining a model which s ill unc ions ully au oma ic:
•
Mul i- ask Lea ning: The backbone o he LSTM model could be ex ended o simul aneously
p edic he mo phokine ic ime poin s (e.g.,
2
,
3
, e c.) alongside ea u e ex ac ion. This
could be achie ed by adding an auxilia y loss unc ion o classi y each ame’s de elopmen al
s age, s ee ing he model o ocus e en mo e on he mo phokine ic s age o he emb yo.
•
Au oma ic Fea u e Ex ac ion: An al e na i e would be o assess he model’s abili y
o au oma ically p edic he mo phokine ic s age and use hese p edic ions o assemble he
ea u e ec o om sec ion 4.1 as inpu o he Machine Lea ning Model. As discussed in
chap e 2, p e ious wo k sugges s ha classi ica ion accu acies o up o 88% can be achie ed
o ame-wise de elopmen al p edic ion. I would be in e es ing o compa e his au oma ic
pipeline o manual anno a ions in e ms o bo h pe o mance and consis ency.
These s a egies could help b idge he pe o mance gap be ween ully au oma ed models and hose
elying on manual anno a ions.
7.3 Missing Me ada a
The cu en da ase lacks key me ada a ha could enhance model pe o mance - speci ically,
e iliza ion me hod and pa ien age.
Including whe he an emb yo was e ilized ia ICSI is impo an , as i in luences he iming o ea ly
de elopmen . Wi hou his in o ma ion, he model may misin e p e na u ally as de elopmen , an
indica o o high de elopmen al po en ial, when i may simply be due o he ICSI p ocedu e.
40

Chap e 7. Discussion and Fu u e Wo k
Simila ly, pa ien age is a well-es ablished ac o a ec ing blas ocys o ma ion a es, wi h ad anced
ma e nal age associa ed wi h lowe de elopmen al po en ial. P o iding he model wi h age da a would
allow i o con ex ualize emb yonic de elopmen pa e ns mo e accu a ely, po en ially imp o ing
bo h p edic ion accu acy and clinical applicabili y.
7.4 Labelling p ocess
To imp o e p edic i e accu acy, he labelling p ocess should in eg a e he Ga dne sco ing sys em,
which explici ly e alua es blas ocys expansion as well as he quali y o he ICM and TE. Wi h he
cu en app oach (Appendix B), emb yo usabili y is in e ed om e ospec i e clinical ou comes,
wi hou di ec e e ence o he Ga dne sco es - in oducing po en ial label noise due o he a iabili y
in clinical decision-making.
Using he Ga dne sco e as a labelling a ge would p o ide a mo e s anda dized and biologically
g ounded signal o aining. Howe e , his equi es manual anno a ion by expe emb yologis s,
which is bo h ime-in ensi e and esou ce-demanding.
A p omising s a egy, as demons a ed by he STEM and STEM+ models in [
22
], is o decouple he
p edic ion ask in o wo s ages: (1) p edic ing whe he an emb yo will each he blas ocys s age,
and (2) classi ying he quali y o he esul ing blas ocys . This sepa a ion could be implemen ed ia
wo independen models o a mul i ask a chi ec u e wi h dual ou pu s.
Reaching he blas ocys s age is ypically easie o anno a e and less suscep ible o subjec i i y, as
i can be de e mined isually wi h ela i e con idence. In con as , g ading blas ocys quali y is
inhe en ly mo e subjec i e, making i mo e p one o in e -obse e a iabili y. By spli ing hese
asks, he lea ning signal becomes mo e speci ic and less noisy, which could help he model con e ge
mo e e ec i ely.
Mo eo e , his sepa a ion has p ac ical ad an ages in clinical se ings. I enhances in e p e abili y
o he model ou pu , especially impo an in cases wi h limi ed emb yos, whe e e en low-quali y
blas ocys s would be used, since hey may esul in success ul p egnancies.
7.5 Enhancing he Da ase Size
As discussed in sec ion 3.2, emb yos ha we e disca ded by clinicians a 72 hou s pos - e iliza ion
we e excluded om he da ase . Consequen ly, he model was ained solely on emb yos ha
appea ed iable a he 72-hou ma k. This in oduces a selec ion bias, po en ially limi ing he
model’s abili y o gene alize o emb yos deemed low-quali y a ha s age. In es iga ing he model’s
beha io on such cases emains an impo an di ec ion o u u e wo k. Howe e , his is inhe en ly
challenging, as disca ded emb yos we e no cul u ed u he and hus lack de elopmen al ou come
labels, making accu a e labelling di icul . No ably, he la ges da ase o da e, assembled by Lassen
e al. [
20
], included disca ded emb yos in he aining p ocess. This app oach imp o ed he model’s
abili y o dis inguish be ween emb yos sui able o ans e and hose likely o be disca ded, helping
o mi iga e bias owa d high-quali y emb yos wi h known implan a ion ou comes.
In addi ion, inc easing he da ase size could signi ican ly imp o e he model’s gene alizabili y. As
o No embe 2024, app oxima ely 10% o he da a was una ailable due o a echnical issue wi h
he hospi al connec ion. Also, new da a con inues o be gene a ed, as he clinic emains ac i e in
conduc ing IVF p ocedu es.
41
Chap e 7. Discussion and Fu u e Wo k
Fu he mo e inco po a ing da a om o he clinics, would enable b oade e alua ion o he model’s
gene alizabili y ac oss di e en clinical en i onmen s. Decoupling he p edic ion o blas ocys
o ma ion om quali y assessmen , as discussed p e iously, also enables he use o public da ase s
such as he one om [
28
], which includes anno a ions o blas ocys o ma ion bu lacks quali y
g ading. This would enable a s anda dized assessmen ac oss a ious s udies and models.
7.6 Assessing o he models
The da ase used in his wo k is compa able in size o ha o [
22
], which is he la ges used o he
classi ica ion ask a hand, p o iding a solid ounda ion o explo ing mo e ecen and ad anced
model a chi ec u es. App oaches such as Video T ans o me s o I3D ne wo ks o e he po en ial o
cap u e mo e complex empo al dynamics om ime-lapse emb yo de elopmen sequences.
Mo eo e , he public a ailabili y o he STEM model code o e s an oppo uni y o di ec compa ison
wi h he models de eloped in his s udy. E alua ing he STEM model on his da ase would be an
impo an assessmen o u u e wo k.
42
Chap e 8
Conclusions
In his Mas e ’s Thesis, a new da ase o emb yo ime-lapse sequences was assembled o ain da a-
d i en AI models aimed a assis ing clinicians in p edic ing he likelihood o success ul blas ocys
de elopmen .
The bes -pe o ming model, based on E icien Ne B0 and enhanced wi h mo phological anno a ions
h ough a ea u e usion mechanism, achie ed a balanced accu acy o 77.4%. In pa allel, a ully
au oma ed ideo-based model inco po a ing an LSTM ne wo k eached a balanced accu acy o
75.3%, demons a ing compe i i e pe o mance wi hou elying on clinical inpu . I was shown ha
he human-anno a ed mo phokine ics a e a e y s ong p edic o o blas ocys de elopmen , and
ully au oma ed models based on images we e no able o ma ch hei pe o mance comple ely.
Assembling and labelling he da ase posed signi ican challenges. None heless, he dis ibu ion
o labels closely e lec s na u al blas ocys o ma ion a es, sugges ing a ealis ic and easonable
labelling p ocess. Wi h o e 10,000 emb yo ime-lapse sequences and app oxima ely 5 million images,
his da ase is one o he la ges o da e used o he ask o blas ocys o ma ion p edic ion.
Compa ed o o he s udies using su icien ly la ge da ase s, he pe o mance me ics achie ed a e on
pa wi h, and in some cases exceed, cu en s a e-o - he-a esul s. Howe e , di ec compa ison
emains di icul due o di e ences in da ase s and me hodologies. S ill, he esul s suppo he
alidi y o bo h he labelling p ocess and da ase assembly.
As he i s s udy using his da ase , he ou comes a e encou aging. The G ad-CAM isualiza ions
u he suppo he model’s eliabili y, showing ocused a en ion on he emb yos a he han
su ounding a i ac s, sugges ing limi ed bias. Howe e , mo e wo k and clinical in eg a ion has o
be done o ensu e he model can be used as a clinical suppo ool in he con ex o IVF- ea men s
and o ensu e ha e hical ai ness and gene aliza ion a e achie ed.
This wo k lays a s ong ounda ion o u u e esea ch on p edic ing high-quali y blas ocys de el-
opmen . In combina ion wi h he ecommenda ions discussed in chap e 7, e en be e esul s a e
easible. The de eloped models a e in ended o u u e deploymen in eal-wo ld clinical se ings
and a e also planned o public elease - po en ially becoming he i s publicly a ailable models o
his ask.
43
Bibliog aphy
[1]
Ashley M. Eskew and Emily S. Jungheim. A his o y o de elopmen s o imp o e in i o
e iliza ion. Missou i Medicine, 114(3):156–159, May-Jun 2017.
[2] N. Bo umandnia, H. Ala i Majd, N. Khadembashi, and H. Alaii. Wo ldwide end analysis o
p ima y and seconda y in e ili y a es o e pas decades: A c oss-sec ional s udy. In e na ional
Jou nal o Rep oduc i e BioMedicine, 20(1):37–46, Feb 2022.
[3]
Alexande R.P. Walke , Be y F. Walke , and Fa ima Adam. Nu i ion, die , physical ac i i y,
smoking, and longe i y: F om p imi i e hun e -ga he e o p esen passi e consume —how a
can we go? Nu i ion, 19(2):169–173, 2003.
[4]
Xiao eng Ye, Xiaoxia Song, Sihang Zhou, Guoqing Chen, and Liping Wang. Associa ion be ween
combined heal hy li es yles and in e ili y: a c oss-sec ional s udy in us ep oduc i e-aged
women. BMC Public Heal h, 25(1):153, 2025.
[5]
And ea C. Go e, Vale ie A. Chappell, Suzanne E. Fen on, Jodi A. Flaws, Angel Nadal, Gail S.
P ins, Jo ma Toppa i, and R. Thomas Zoelle . EDC-2: The Endoc ine Socie y’s Second
Scien i ic S a emen on Endoc ine-Dis up ing Chemicals. Endoc ine Re iews, 36(6):E1–E150,
Dec 2015. Epub 2015 No 6.
[6]
B adley J. Van Voo his. In i o e iliza ion. New England Jou nal o Medicine, 356(4):379–386,
2007.
[7]
Pa izia Rubino, Paola Viganò, Alice Luddi, and Paola Piomboni. The icsi p ocedu e om pas
o u u e: a sys ema ic e iew o he mo e con o e sial aspec s. Human Rep oduc ion Upda e,
22(2):194–227, 11 2015.
[8]
Ki s ine Ki kegaa d, Aishling Ahls öm, Hans Jakob Inge sle , and Tho i Ha da son. Choosing
he bes emb yo by ime lapse e sus s anda d mo phology. Fe ili y and S e ili y, 103(2):323–332,
2015.
[9]
J. Conaghan, A.A. Chen, S.P. Willman, K. I ani, P.E. Chene e, R. Boos an a , V.L. Bake ,
G.D. Adamson, M.E. Abusie , M. G akha ia, K.E. Loewke, and S. Shen. Imp o ing emb yo
selec ion using a compu e -au oma ed ime-lapse image analysis es plus day 3 mo phology:
esul s om a p ospec i e mul icen e ial. Fe ili y and S e ili y, 100(2):412–419.e5, 08 2013.
Epub 2013 May 28.
[10]
M. Salih, C. Aus in, R. R. Wa y, e al. Emb yo selec ion h ough a i icial in elligence
e sus emb yologis s: a sys ema ic e iew. Human Rep oduc ion Open, 2023(3):hoad031, 2023.
Published 2023 Aug 15.
[11]
Connie C. Wong, Ke in E. Loewke, Nancy L. Bosse , Ba y Beh , Ch is ophe J. De Jonge,
45

Bibliog aphy
Thomas M. Bae , and Renee A. Reijo Pe a. Non-in asi e imaging o human emb yos be o e
emb yonic genome ac i a ion p edic s de elopmen o he blas ocys s age. Na u e Bio echnology,
28(10):1115–1121, 2010.
[12]
Kuan-Chong Lan, Feng-Jung Huang, Yu-Ching Lin, Fu-Tsai Kung, Chian-Huey Hsieh, Hsin-Wen
Huang, Ping-Hui Tan, and Szu-Yuan Chang. The p edic i e alue o using a combined z-sco e
and day 3 emb yo mo phology sco e in he assessmen o emb yo su i al on day 5. Human
Rep oduc ion, 18(6):1299–1306, Jun 2003.
[13]
Hui Zou, Jason M Kempe , Elizabe h R Hammond, Fei Xu, Guanghui Liu, Li Xue, Xi Bai,
Hong Liao, Shuang Xue, Shuai Zhao, Lei Xia, John Sco , Vanessa Chapple, Mo A nan, Dean E
Mo beck, Ben Willem J Mol, Ying Liu, and Rui Wang. Blas ocys quali y and ep oduc i e
and pe ina al ou comes: a mul ina ional mul icen e obse a ional s udy. Human Rep oduc ion,
38(12):2391–2399, Dec 2023.
[14]
Na han H Ng, Julian McAuley, Julian A Gingold, Nina Desai, and Zacha y C Lip on. P edic ing
emb yo mo phokine ics in ideos wi h la e usion ne s & dynamic decode s, 2018.
[15]
Reza Mo adi Rad, Pa aneh Saeedi, Jason Au, and Jon Ha elock. Blas ome e cell coun ing and
cen oid localiza ion in mic oscopic images o human emb yo. 2018 IEEE 20 h In e na ional
Wo kshop on Mul imedia Signal P ocessing (MMSP), pages 1–6, 2018.
[16]
Chongwei Wu, Wei Yan, Hong u Li, Jiaxin Li, Hongkai Wang, Shijie Chang, Tao Yu, Ying Jin,
Chao Ma, Yahong Luo, Dongxu Yi, and Xi an Jiang. A classi ica ion sys em o day 3 human
emb yos using deep lea ning. Biomedical Signal P ocessing and Con ol, 70:102943, 2021.
[17]
Pegah Khos a i, Ehsan Kazemi, Qiansheng Zhan, Jonas E. Malms en, Ma co Toschi, Pan elis
Zisimopoulos, Alexand os Siga as, S ua La e y, Lee A. D. Coope , C is ina Hickman, Ma cos
Mesegue , Ze Rosenwaks, Oli ie Elemen o, Nikica Zanino ic, and Iman Haji asouliha. Deep
lea ning enables obus assessmen and selec ion o human blas ocys s a e in i o e iliza ion.
NPJ Digi al Medicine, 2:21, 04 2019. eCollec ion 2019.
[18]
M. Ve Milyea, J.M.M. Hall, S.M. Diakiw, A. Johns on, T. Nguyen, D. Pe ugini, A. Mille ,
A. Picou, A.P. Mu phy, and M. Pe ugini. De elopmen o an a i icial in elligence-based
assessmen model o p edic ion o emb yo iabili y using s a ic images cap u ed by op ical
ligh mic oscopy du ing i . Human Rep oduc ion, 35(4):770–784, 04 2020.
[19]
S. M. Diakiw, J. M. M. Hall, M. Ve Milyea, A. Y. X. Lim, W. Quangkananu ug, S. Chan-
cham oen, B. Bankowski, R. S ones, A. S o , A. Mille , G. Adaniya, R. an Tol, R. Hanson,
J. Aizpu ua, L. Gia dini, A. Johns on, T. Van Nguyen, M. A. Dakka, D. Pe ugini, and M. Pe-
ugini. An a i icial in elligence model co ela ed wi h mo phological and gene ic ea u es
o blas ocys quali y imp o es anking o iable emb yos. Rep oduc i e Biomedicine Online,
45(6):1105–1117, 2022.
[20]
Jacob Theilgaa d Lassen, Mikkel Fly K agh, Jens Rimes ad, Ma in Nygå d Johansen, and
Jø gen Be n sen. De elopmen and alida ion o deep lea ning based emb yo selec ion ac oss
mul iple days o ans e . Scien i ic Repo s, 13(1):4235, 2023.
[21]
Joao Ca ei a and And ew Zisse man. Quo adis, ac ion ecogni ion? a new model and he
kine ics da ase , 2018.
[22]
Qiuyue Liao, Qi Zhang, Xue Feng, Haibo Huang, Haohao Xu, Baoyuan Tian, Jihao Liu, Qihui
Yu, Na Guo, Qun Liu, Bo Huang, Ding Ma, Jihui Ai, Shugong Xu, and Kezhen Li. De elopmen
46
Bibliog aphy
o deep lea ning algo i hms o p edic ing blas ocys o ma ion and quali y by ime-lapse
moni o ing. Communica ions Biology, 4(1):415, 03 2021.
[23]
G. Co icchio, G. Fio en ino, G. Nico a, R. Sciajno, F. Ca ale a, R. Bellazzi, S. Ga agna,
A. Bo ini, and M. Zucco i. Cy oplasmic mo emen s o he ea ly human emb yo: imaging and
a i icial in elligence o p edic blas ocys de elopmen . Rep od Biomed Online, 42(3):521–528,
03 2021. Epub 2020 Dec 24.
[24]
Manoj Kuma Kanakasabapa hy, P udh i Thi umala aju, Cha les L Bo mann, Ragha Gup a,
Rohan Pooniwala, Heman h Kandula, I ene Sou e , I ene Dimi iadis, and Hadi Sha iee. Deep
lea ning media ed single ime-poin image-based p edic ion o emb yo de elopmen al ou come
a he clea age s age, 2020.
[25]
T. Chen e al. Au oma ing blas ocys o ma ion and quali y p edic ion in ime-lapse imaging
wi h adap i e key ame selec ion. In L. Wang, Q. Dou, P.T. Fle che , S. Speidel, and S. Li,
edi o s, Medical Image Compu ing and Compu e Assis ed In e en ion - MICCAI 2022, olume
13434 o Lec u e No es in Compu e Science, pages 470–480. Sp inge , Cham, 2022.
[26]
As id Zeman, Anne-So ie Mae en, Annemie Mengels, Lie Fong Sha on, Ca l Spiessens, and
Hans Op de Beeck. Deep lea ning o human emb yo classi ica ion a he clea age s age (day
3). In Albe o Del Bimbo, Ri a Cucchia a, S an Scla o , Gio anni Ma ia Fa inella, Tao Mei,
Ma co Be ini, Hugo Jai Escalan e, and Robe o Vezzani, edi o s, Pa e n Recogni ion. ICPR
In e na ional Wo kshops and Challenges, pages 278–292, Cham, 2021. Sp inge In e na ional
Publishing.
[27]
Kanak Kalyani and Pa ag S. Deshpande. A deep lea ning model o p edic ing blas ocys
o ma ion om clea age-s age human emb yos using ime-lapse images. Scien i ic Repo s,
14(1):28019, 2024.
[28]
T is an Gomez, Magalie Feyeux, Jus ine Boulan , Nicolas No mand, Lau en Da id, Pe ine
Paul-Gillo eaux, Thomas F éou , and Ha old Mouchè e. A ime-lapse emb yo da ase o
mo phokine ic pa ame e p edic ion. Da a in B ie , 42:108258, 2022.
[29]
Phillip A. Romanski, Ashley Aluko, Pie o Bo ole o, Rony T. Elias, and Ze Rosenwaks. Age-
speci ic blas ula ion a es in emb yo c yop ese a ion cycles yielding a c yop ese ed blas ocys .
Fe ili y and S e ili y, 116(1):e11–e12, 2021.
[30]
R. Sain e-Rose, C. Pe i , L. Dijols, C. F apsauce, and F. Gue i . Ex ended emb yo cul u e is
e ec i e o pa ien s o an ad anced ma e nal age. Scien i ic Repo s, 11(1):13499, 2021.
[31]
Lisbe Van Landuy , Anick De Vos, Hube Jo is, G e a Ve heyen, Paul De oey, and And é Van
S ei eghem. Blas ocys o ma ion in in i o e iliza ion e sus in acy oplasmic spe m injec ion
cycles: In luence o he e iliza ion p ocedu e. Fe ili y and S e ili y, 83(5):1397–1403, 2005.
[32]
Ba ba a Speye , Helen O’Neill, Wael Saab, S i idya Seshad i, Suzanne Cawood, Ca leen Hea h,
Ma hew Gaun , and Paul Se hal. In assis ed ep oduc ion by i o icsi, he a e a which
emb yos de elop o he blas ocys s age is in luenced by he e iliza ion me hod used: a spli
i /icsi s udy. Jou nal o Assis ed Rep oduc ion and Gene ics, 36(4):647–654, 2019.
[33]
Ma iana Nicolielo, Ca he ine Jacobs, And ea Belo, Ana Paula Reis, Rena a E be elli, Fabiana
Mendez, Ma ina Fanelli, Li ia C emonesi, Paulo Cesa Se a ini, Edua do L.A. Mo a, Aline R.
Lo enzon, and Jose Robe o Aleg e i. Emb yo cul u e in ime-lapse sys em p o ides be e
47
Bibliog aphy
a es o blas ocys o ma ion, dec eases emb yo de elopmen a es a e compa ed o adi ional
iple-gas cul u e sys em. Fe ili y and S e ili y, 112(3):e125–e126, 2019.
[34]
Alexand a J Ke mack, I ina Fesenko, Da id R Ch is ensen, Ka e L Pa y, Philippa Lowen,
Susan J Wells ead, Sco F Ha is, Philip C Calde , Nicholas S Macklon, and F anchesca D
Hough on. Incuba o ype a ec s human blas ocys o ma ion and emb yo me abolism: a
andomized con olled ial. Human Rep oduc ion, 37(12):2757–2767, Oc 2022.
[35]
Ga y B adski. Openc : Open sou ce compu e ision lib a y.
h ps://openc .o g/
, 2000.
Accessed: 2025-03-21.
[36]
Samuel Smi h. py esse ac : Py hon- esse ac .
h ps://gi hub.com/madmaze/py esse ac
,
2023. Accessed: 2025-03-21.
[37] Leo B eiman. Random o es s. Machine Lea ning, 45(1):5–32, 2001.
[38]
Yann LeCun, Yoshua Bengio, and Geo ey Hin on. Deep lea ning. Na u e, 521(7553):436–444,
2015.
[39]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep esidual lea ning o image
ecogni ion, 2015.
[40] Ilya Loshchilo and F ank Hu e . Decoupled weigh decay egula iza ion, 2019.
[41]
Ze Liu, Yu ong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, S ephen Lin, and Baining
Guo. Swin ans o me : Hie a chical ision ans o me using shi ed windows, 2021.
[42] Ashish Vaswani, Noam Shazee , Niki Pa ma , Jakob Uszko ei , Llion Jones, Aidan N. Gomez,
Lukasz Kaise , and Illia Polosukhin. A en ion is all you need, 2023.
[43]
Mingxing Tan and Quoc V. Le. E icien ne : Re hinking model scaling o con olu ional neu al
ne wo ks, 2020.
[44]
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Ch is oph Feich enho e , T e o Da ell, and Saining
Xie. A con ne o he 2020s, 2022.
[45]
Saining Xie, Ross Gi shick, Pio Dollá , Zhuowen Tu, and Kaiming He. Agg ega ed esidual
ans o ma ions o deep neu al ne wo ks, 2017.
[46]
Ch is ian Szegedy, Vincen Vanhoucke, Se gey Io e, Jon Shlens, and Zbigniew Wojna. Re hinking
he incep ion a chi ec u e o compu e ision. In P oceedings o he IEEE Con e ence on
Compu e Vision and Pa e n Recogni ion (CVPR), pages 2818–2826, 2016.
[47]
Gao Huang, Zhuang Liu, Lau ens an de Maa en, and Kilian Q. Weinbe ge . Densely connec ed
con olu ional ne wo ks, 2018.
[48]
Ka en Simonyan and And ew Zisse man. Ve y deep con olu ional ne wo ks o la ge-scale
image ecogni ion, 2015.
[49]
Jia Deng, Wei Dong, Richa d Soche , Li-Jia Li, Kai Li, and Li Fei-Fei. Imagene : A la ge-scale
hie a chical image da abase. In 2009 IEEE Con e ence on Compu e Vision and Pa e n
Recogni ion, pages 248–255, 2009.
[50]
Jason Yosinski, Je Clune, Yoshua Bengio, and Hod Lipson. How ans e able a e ea u es in
deep neu al ne wo ks?, 2014.
48
Bibliog aphy
[51]
Xi Mei, Zongyu Liu, Paul M. Robson, Benjamin Ma inelli, MingDe Huang, Apu a Doshi,
Adam Jacobi, Congyu Cao, Ke in E. Link, Tian Yang, Yiqiu Wang, Hayi G eenspan, Tessa
Deye , Zahi A. Fayad, and Yang Yang. Radimagene : An open adiologic deep lea ning esea ch
da ase o e ec i e ans e lea ning. Radiology: A i icial In elligence, 4(5):e210315, 2022.
[52]
Ramp asaa h R. Sel a aju, Michael Cogswell, Abhishek Das, Ramak ishna Vedan am, De i
Pa ikh, and Dh u Ba a. G ad-cam: Visual explana ions om deep ne wo ks ia g adien -
based localiza ion. In 2017 IEEE In e na ional Con e ence on Compu e Vision (ICCV), pages
618–626, 2017.
[53]
Junyoung Chung, Cagla Gulceh e, KyungHyun Cho, and Yoshua Bengio. Empi ical e alua ion
o ga ed ecu en neu al ne wo ks on sequence modeling, 2014.
[54]
Jiquan Ngiam, Adi ya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and And ew Ng.
Mul imodal deep lea ning. pages 689–696, 01 2011.
[55]
Sepp Hoch ei e and Jü gen Schmidhube . Long sho - e m memo y. Neu al Compu .,
9(8):1735–1780, No embe 1997.
[56]
F. Ped egosa, G. Va oquaux, A. G am o , V. Michel, B. Thi ion, O. G isel, M. Blondel,
P. P e enho e , R. Weiss, V. Dubou g, J. Vande plas, A. Passos, D. Cou napeau, M. B uche ,
M. Pe o , and E. Duchesnay. Sciki -lea n: Machine lea ning in Py hon. Jou nal o Machine
Lea ning Resea ch, 12:2825–2830, 2011.
[57]
William Falcon and The PyTo ch Ligh ning eam. Py o ch ligh ning.
h ps://gi hub.com/
Ligh ning-AI/ligh ning
, 2019. The ligh weigh PyTo ch w appe o high-pe o mance AI
esea ch. Scale you models, no he boile pla e.
[58]
Jason Ansel, Edwa d Yang, Ho ace He, Na alia Gimelshein, Animesh Jain, Michael Voznesensky,
Bin Bao, Pe e Bell, Da id Be a d, E geni Bu o ski, Gee a Chauhan, Anjali Chou dia, Will
Cons able, Alban Desmaison, Zacha y DeVi o, Elias Ellison, Will Feng, Jiong Gong, Michael
Gschwind, B ian Hi sh, She lock Huang, Kshi eej Kalamba ka , Lau en Ki sch, Michael Lazos,
Ma io Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Be Mahe , Yunjie Pan,
Ch is ian Puh sch, Ma hias Reso, Ma k Sa ou im, Ma cos Yukio Si aichi, Helen Suk, Michael
Suo, Phil Tille , Eikan Wang, Xiaodong Wang, William Wen, Shun ing Zhang, Xu Zhao,
Ke en Zhou, Richa d Zou, Aji Ma hews, G ego y Chanan, Peng Wu, and Soumi h Chin ala.
Py o ch 2: Fas e machine lea ning h ough dynamic py hon by ecode ans o ma ion and
g aph compila ion. In P oceedings o he 29 h ACM In e na ional Con e ence on A chi ec u al
Suppo o P og amming Languages and Ope a ing Sys ems, Volume 2 (ASPLOS ’24). ACM,
Ap il 2024.
[59]
Lucia U celay, Daniel Hinjos, Pablo A. Ma in-To es, Ma a Gonzalez, Ma a Mendez, Sal a
Cí ico, Se gio Ál a ez Napagao, and Da io Ga cia-Gasulla. Explo ing he ole o explainabili y
in ai-assis ed emb yo selec ion, 2023.
49

Appendix B
Emb yo Selec ion
On he ollowing page, he lowcha o emb yo labelling is shown. Each Emb yoID wi h alid
anno a ions unde goes his p ocess o de e mine whe he i should be included in he da ase . The
decision is based solely on he anno a ions associa ed wi h each emb yo. Due o he da ase ’s la ge
size and he long download imes, i was no easible o download all ideos i s and hen decide
based on mo phology o he p esence o ames a speci ic ime poin s.
The possible ou comes o he decision p ocess a e: Disca d,T ue Label, o False Label. Each ci cle in
he lowcha shows, in i alic, he numbe o emb yos alling in o ha ca ego y. The decisions e lec
he clinical p ocedu es ollowed a he hospi al, wi h each case e alua ed by a p ac icing clinician.
I should also be no ed ha app oxima ely 100 emb yos we e manually disca ded due o speci ic
issues (e.g., emp y wells, occluded emb yos). Howe e , manual inspec ion o e e y emb yo was no
easible ac oss he ull da ase .
57
All Emb yoIDs on he
emb yoscope
No
Yes
Mos ecen
anno a ion ime is
a leas 72 hou s
Disca d
No
Fi s image
o sequence is no la e
han 5 hou s
Disca d
yes
no
Dead Label?
yes
no
Emb yo has B,
EB o HB label
yes
no B, EB
and HB no
p esen ?
F eeze T ans e
A oid
Unknown/Undecided
406
Fa e?
1. Bad
Emb yo
Blas ocys
No
854
abno mal
2
unknown
PN numbe ?
9. A oided
abno mal
emb yos
326
8. A oided
no mal
emb yos
2583
7. A oided
Unknown
Emb yos
Yes
6. Emb yo is a
clea age s age
ans e , g ound
u h no known,
Disca d
837
yes
no
EB Label
(expanded)?
2
unknown
abno mal
PN numbe ?
17. Exp.
Blas . wi h
no mal PN
148
19. Exp.
Blas wi h
abno mal PN
14
18. Exp. Blas
wi h unknown
PN
16. A oid.
Blas . no
eaching exp.
s age
432
a oid
unknown
eeze / ans e
undecided
Fa e?
2 o unknown
3, 4, 5, ..
1
PN numbe ?
13.
Good emb yo
Blas ocys
T ue
4184
14. Good
emb yo wi h
abno mal PN
108
no
yes
EB label
(expanded)?
2. incipien
blas . ha
died
1
Emb yo Labelling Decision T ee
unknown
abno mal
2PN numbe ?
10.
Comple ely
Unknown
Emb yo
12. abno m.
non- a e
emb yo
118
11. No mal
non- a e
emb yo
1296
2
unknown
abno mal
3. exp. blas .
ha died wi h
no mal PN
16
4. exp. blas .
ha died wi h
unknown PN
5. exp. blas .
ha died wi h
abno mal PN
PN numbe ?
yes no
EB Label
(expanded)?
15a.
Expanded no-
a e
Blas ocys
136
15b.
Incipien non-
a e
Blas ocycs
270
Abno mal PN
numbe sugges a
mis ake in ou come
o PN numbe
The emb yos did no
each a Blas ocys
s age, and we e
a oided.
Gene ally ea as bad
examples, e en ho his
inlcudes some doc o s
bias, since he eason
could be a p elimina y
doc o 's decision.
Abno mal PN numbe
sugges s bad quali y,
he e o e used as False
samples
mo phologically
no good, li le
cells, mal o med,
e c.
missing da a o he a e, quali y
o he blas ocys can no be
alida ed
Disca d Emb yos
False Label
T ue Label
PN=1: Emb yos
can each
blas ocys s age
and a e alid
13a. Good
emb yo wi h
PN=1
108
c ossed
ou No samples in da ase
Labeling Ou comes
Appendix C
Emb yo Well C opping
The emb yo well c opping echnique elies on he Hough Ci cle T ans o m o accu a ely de ec he
ci cula well s uc u e in each ame. To ensu e obus de ec ion, a se ies o image p ep ocessing
s eps is applied. Based on empi ical e alua ion, he ollowing pipeline yielded he mos eliable
esul s. All me hods desc ibed he e we e implemen ed using he py hon implemen a ion o he
OpenCV lib a y [35].
Figu e C.1:
S ep-by-s ep p ep ocessing o Hough Ci cle de ec ion. F om le o igh : he o iginal image;
e osion il e o enhance con as be ween he well and he backg ound; applica ion o he Sobel
il e o highligh edges and supp ess in e nal ex u es; u he e osion o emo e small a i ac s
such as emb yo ou lines; Gaussian blu ing o educe noise and smoo h he image o mo e
s able ci cle de ec ion.
A e p ep ocessing, he Hough Ci cle T ans o m is applied o de ec he well. De ec ion pa ame e s
a e adjus ed based on he machine ype (Emb yoScope+ o Emb yoScope-D) and hei co esponding
image esolu ion (800×800 o 500×500 pixels). The minimum and maximum ci cle adii a e uned
acco dingly o es ic de ec ion o plausible well sizes.
A key pa ame e in he Hough Ci cle T ans o m is he ci cle de ec ion sensi i i y
pa am2
. I no
ci cle is ini ially de ec ed, he sensi i i y h eshold is g adually lowe ed un il ci cles a e ound o a
minimum alue is eached, allowing mo e pe missi e de ec ion in challenging cases.
In mos ins ances, mul iple candida e ci cles a e iden i ied. To ob ain a s able and consis en
es ima e, he inal well posi ion is compu ed as he a e age o all de ec ed ci cle cen e s and adii.
59
Appendix C. Emb yo Well C opping
Figu e C.2:
De ec ed ci cles (le ) and he inal a e aged ci cle ( igh ), which is used as he es ima ed well.
Finally, he image is c opped o he egion co esponding o he es ima ed well. All a eas ou side
he ci cula bounda y a e masked in black. The esul ing image is hen esized o ma ch he o iginal
dimensions, p ese ing spa ial consis ency ac oss he da ase .
60