MicroSplit: Semantic Unmixing of Fluorescent Microscopy Data

Mic oSpli : Seman ic Unmixing o Fluo escen Mic oscopy Da a
Ashesh Ashesh1, Fede ico Ca a a1, Igo Zuba e 1, Ve a Galino a1, Melisande C o 1,
Melissa Pezzo i2, Daozheng Gong3, F ancesca Casag ande1, Elisa Colombo1,
S e ania Giussani1, Elena Res elli1, Eugenia Camma o a1, Juan Manuel Ba aglio i1,
Nikolai Klena1, Moises Di San e2, Raghabend a Adhika i4, Daniel Feliciano4, Gaia Pigino1,
Elena Ta e na1, Oli e Ha schni z1, Nicola Maghelli1, No be Sche e 3,
Damian Edwa d Dalle Noga e1, Jo an Deschamps1, F ancesco Pasqualini2, Flo ian Jug1*
1Fondazione Human Technopole, V.le Ri a Le i-Mon alcini, 20157, Milan, I aly.
2Uni e si y o Pa ia, Co so S ada Nuo a, 65, 27100, Pa ia, I aly.
3Uni e si y o Chicago, 5801 S Ellis A e, 60637, Chicago, USA.
4HHMI/Janelia Resea ch Campus, 19700 Helix D i e, Ashbu n, 20147, VA, USA.
Abs ac
Fluo escence mic oscopy, a key d i e o p og ess in he li e sciences, aces limi a ions due o he
mic oscope’s op ics, luo opho e chemis y, and pho on exposu e limi s, necessi a ing ade-o s in
imaging speed, esolu ion, and dep h. He e, we in oduce Mic oSpli , a compu a ional mul iplexing ech-
nique based on deep lea ning ha allows mul iple cellula s uc u es o be imaged in a single luo escen
channel and hen unmix hem by compu a ional means, allowing as e imaging and educed pho on
exposu e. We show ha Mic oSpli e icien ly sepa a es up o ou supe imposed noisy s uc u es in o
dis inc denoised luo escen image channels. Fu he mo e, using Va ia ional Spli ing Encode -Decode
(VSE) ne wo ks, ou app oach can sample di e se p edic ions om a ained pos e io o solu ions. The
di e si y o hese samples scales wi h he unce ain y in a gi en inpu , allowing us o es ima e he ue
p edic ion e o s by compu ing he a iabili y be ween pos e io samples. We demons a e he obus -
ness o Mic oSpli ne wo ks, which a e ained o each spli ing ask a hand, ac oss a ious da ase s
and noise le els and show i s u ili y o image mo e, o image as e , and o imp o e downs eam analy-
sis. We p o ide Mic oSpli along wi h all associa ed aining and e alua ion da ase s as open esou ces,
enabling li e scien is s o immedia ely bene i om he po en ial o compu a ional mul iplexing and hus
help accele a e he a e o hei scien i ic disco e y p ocess.
Keywo ds: deep lea ning, luo escence mic oscopy, compu a ional mul iplexing, seman ic channel unmixing,
a ia ional in e ence
1 In oduc ion
Fluo escence mic oscopy is an essen ial ool o explo ing s uc u es and dynamics wi hin cells, issues, and
o ganisms. Mul iplexed acquisi ion echniques in ol ing mul iple luo opho es, each agged o a di e en
cellula s uc u e, a e used o acqui e mul ichannel image da a. Howe e , o e lap in luo opho e exci a ion
spec a imposes p ac ical limi s on he numbe o luo opho es ha can be used in a biological sample (see
Figu e 1a) wi hou p oblems such as c oss- alk o bleed- h ough [1]. E en i a sui able se o luo escen
labels ha e been chosen, he sequen ial na u e o mul iplexed acquisi ions equi es mul iple exposu es o he
sample, which educes he pho on budge a ailable o o he pu poses [2,3] and also limi s he maximum
sampling equency in li e-cell imaging scena ios. A limi ed pho on budge can be add essed by educing
he pho on exposu e pe acquisi ion, which in u n will lead o mo e noisy images ha will be ha de o
analyze [3–5].
In his wo k, we desc ibe a compu a ional mul iplexing app oach, Mic oSpli , which add esses he limi-
a ions ou lined abo e. We p opose labeling and imaging mul iple biological s uc u es in a single luo escen
1
Fig. 1 Seman ic unmixing o luo escence mic oscopy da a wi h Mic oSpli . (a) Mul iplexed luo escen imaging
is conduc ed by labeling cellula componen s wi h di e en luo escen ma ke s and imaging one a e ano he in o sepa a e
image channels. (b) We p opose a me hod ha allows mul iple cellula s uc u es o be imaged simul aneously in a single
image channel. The supe imposed s uc u es a e hen spli in o sepa a e channels using an adequa ely ained neu al ne wo k.
(c) Imaging mul iple s uc u es in one go sa es p ecious pho on budge . I is well known ha he o al pho on budge limi s
he capabili ies o ligh mic oscopes (le ). Using con en -awa e denoising me hods, e.g. [3,4], can help epu pose some pho on
budge by acqui ing mo e noisy aw images (middle). Imaging mul iple cellula componen s in a single luo escen channel
sa es addi ional pho on budge ha can hen, o example, be in es ed in o imaging addi ional s uc u es, image mo e gen le,
o image as e ( igh ). (d- ) Exempla y quali a i e esul s o wo-channel spli ing (d), h ee channel spli ing (e), and ou -
channel spli ing ( ). Each panel shows he image channel con aining mul iple s uc u es o be spli (inpu ), inse s o he noisy
a ge channels (Ci, i ∈ {1,2,3,4}) as hey a e used du ing aining ( a ge ), and inse s o he spli channels as p edic ions by
Mic oSpli (p edic ion). No e ha he p edic ions a e no iceable less noisy han he aining da a i sel , which is a key ea u e
o Mic oSpli ’s unsupe ised denoising componen (see also he main ex and Figu e 2).
channel and hen employing ou p oposed me hod o spli supe imposed s uc u es om wi hin his mul i-
s uc u e channel compu a ionally in o sepa a e unmixed image channels (see Figu e 1b). We ecen ly
p esen ed he unde lying me hodological componen s ha can enable such compu a ional mul iplexing in
heo y [6,7] and ha e now c ea ed a p ac ical me hod ha (i) combines he bene i s o bo h app oaches in o
a single amewo k, (ii) enables he p ocessing o olume ic image da a by c ea ing a highly op imized ne -
wo k a chi ec u e o i (see Figu e 1and Table 1), and (iii) we p opose a p ocedu e o assess he calib a ion
o a ained ne wo k and he es ima ion o p edic ion e o s (see Figu e 2and Sec ion 2.3). Mic oSpli is
designed o easy use by mic oscopis s and li e scien is s, e en hose who a e no machine lea ning expe s.
2
Using Mic oSpli allows use s o (i) simul aneously image mo e s uc u es han p e iously possible by
combining up o ou s uc u es in a single channel, o o (ii) image he same numbe o s uc u es bu in
ewe channels and, hence, a educed pho on exposu e (see Figu e 1c). The sa ed pho on budge can hen
be alloca ed owa ds o he objec i es such as as e empo al sampling, be e signal- o-noise a io in aw
acquisi ions, e.g. o image mo e gen le and a oid pho o oxici y, o any combina ion o hese possibili ies.
No e ha each seman ic unmixing ask equi es aining a dedica ed Mic oSpli model. While aining a
uni e sal ounda ion model is concep ually possible, we delibe a ely employ na ow, ask-speci ic models o
a oid ou -o -dis ibu ion issues and o ensu e obus pe o mance on he da a a hand.
Ou expe imen s show ha Mic oSpli is capable o sepa a ing wo, h ee, o e en ou join ly imaged
s uc u es in o denoised unmixed channel p edic ions, e en i he aining da a a e i sel noisy (see Figu e 1d-
). A Mic oSpli model lea ns o unmix supe imposed s uc u es by lea ning om examples (i.e. in a
supe ised way). A he same ime, i co-lea ns o denoise he da a wi hou supe ision (see Figu e 2a).
This means ha i is su icien o use a body o noisy aining da a and Mic oSpli will s ill lea n o p edic
denoised images o each unmixed s uc u e (images labeled “Ta ge ”1in Figu es 1and 2show such noisy
aining da a). These p ope ies make ou me hod s aigh o wa d o apply in p ac ical applica ions, as we
show in de ail in Sec ion 2.
An addi ional and dis inc i e ea u e o he Va ia ional Spli ing Encode -Decode (VSE) ne wo ks we
a e using is hei abili y o gene a e mul iple plausible solu ions o a gi en inpu image [7]. When gi en
inpu s a e unambiguous and hus can be spli wi h high ce ain y, he gene a ed solu ions will closely
esemble each o he . In con as , as he unce ain y in inpu s inc eases, he di e si y among he sampled
solu ions will e lec hese inhe en ambigui ies by becoming mo e di e en om one ano he . We show ha
he ne wo ks we ained a e calib a ed, allowing us o es ima e he o he wise ha d o assess ue e o o
p edic ions, e en in he absence o g ound u h images ha a e no o iously ha d o e en impossible o come
by. Technically, his is enabled by e alua ing he easy- o-compu e in e -sample a iabili y (see Sec ion 2.3).
This ea u e is c ucial because i enables use s o iden i y he pa s o hei da a whe e Mic oSpli p edic ions
may be un eliable due o ambigui ies in he ed inpu s. Such egions can ei he be dis ega ded o subjec ed
o expe e iew, e ec i ely add essing a pe sis en challenge in AI-d i en bioimage analysis– he di icul y
in e alua ing he accu acy o p edic ions [1].
We show he pe o mance o Mic oSpli on 24 2D and 5 3D seman ic unmixing asks, showing ha we
achie e accu a e esul s in a wide ange o noise le els and supe imposed s uc u es, consis en ly leading o
high-quali y denoised single-s uc u e p edic ions.
In addi ion o he abo e-men ioned use cases, we p opose a way o use Mic oSpli o emo e unwan ed
s uc u ed image a i ac s by unmixing hem om he s uc u es in which we a e uly in e es ed. We
showcase his capabili y by emo ing spu ious luo escen punc a om a luo escen mic oscopy da ase (see
Figu e 4and Sec ion 2.4.2).
2 Resul s
As in oduced abo e, Mic oSpli enables mic oscopis s o image mul iple s uc u es ha would ypically be
imaged in sepa a e luo escence channels and ins ead: (i) cap u e hese s uc u es in one channel, and hen
(ii) unmix he supe imposed s uc u es con ained in ha channel by compu a ional means. The aining
p ocedu e we p opose equi es supe ision signals ( a ge images) o each s uc u e o be unmixed (see
Figu e 1b). In he ollowing sec ion, we ou line h ee p ac ical ways o ob ain sui able aining da a and
will use hem in a ious expe imen al se ings o demons a e hei e ec i eness.
2.1 T aining Modes and Requi ed T aining Da a
T adi ionally, da a a e acqui ed using con en ional mul iplexing (see Figu e 1a). Hence, we p opose
T aining Mode I, whe e p e iously eco ded mul iplexed image channels a e used as a ge s o supe -
ised aining o achie e seman ic unmixing o cellula s uc u es (see Figu e 2a). We gene a e mixed inpu
images by pixel-wise summa ion o mul iplexed image channels. These inpu images closely esemble wha
can la e be acqui ed in a single acquisi ion (see Figu e 1b). Al hough his app oach yields high-quali y
aining da a, some di e ences in noise p ope ies and in ensi y a ia ions be ween s uc u es may a ise,
as discussed in Sec ion C.2. Expe imen s using T aining Mode I show consis en ly high seman ic unmixing
pe o mance, as seen in Figu es 1and 2, Table 1, and in Sec ions B.2 and C.2.
1In many supe ised aining se ings, he supe ision da a is e e ed o as “g ound u h”. In his manusc ip we ins ead e e o
he supe ision signals as “Ta ge ” since Mic oSpli can be ained exclusi ely on noisy da a ha is he e o e no g ound u h.
3
(a) Two-channel unmixing asks
Task Da ase Task De ails 2D/3D PSNR Mic oMS-SSIM
C1 C2 C1 C2
I HT-H24 TM-I 3D 38.8 33.8 0.970 0.951
II HT-P23A TM-I 3D 25.1 31.4 0.747 0.932
III HT-P23B TM-I 3D 26.4 21.6 0.847 0.599
IV Pa ia-P24 TM-III, high, 50:50 2D 24.3 29.9 0.682 0.839
V Pa ia-P24 TM-III, high, 66:33 2D 28.2 25.6 0.780 0.696
VI Pa ia-P24 TM-III, high, 84:16 2D 25.2 24.1 0.722 0.623
VII Pa ia-P24 TM-III, mid, 50:50 2D 23.1 24.3 0.673 0.755
VIII Pa ia-P24 TM-III, mid, 66:33 2D 24.0 22.3 0.729 0.678
IX Pa ia-P24 TM-III, mid, 84:16 2D 24.3 22.4 0.735 0.659
X Pa ia-P24 TM-III, low, 50:50 2D 21.9 23.0 0.612 0.684
XI Pa ia-P24 TM-III, low, 66:33 2D 22.9 21.7 0.593 0.564
XII Pa ia-P24 TM-III, low, 84:16 2D 23.3 22.8 0.682 0.663
XIII HT-T24 TM-III 2D 40.3 32.8 0.978 0.951
XIV HT-LIF24 TM-III 2D 32.0 32.9 0.965 0.960
XV Chicago-Sch23 TM-I, C0 s C1 2D 41.3 42.9 0.984 0.993
XVI Chicago-Sch23 TM-I, C0 s C2 2D 38.4 41.1 0.973 0.987
XVII Chicago-Sch23 TM-I, C0 s C3 2D 58.2 41.7 0.998 0.993
XVIII Chicago-Sch23 TM-I, C1 s C2 2D 43.7 44.9 0.995 0.995
XIX Chicago-Sch23 TM-I, C1 s C3 2D 65.4 44.3 1.000 0.996
XX Chicago-Sch23 TM-I, C2 s C3 2D 66.6 43.6 1.000 0.996
(b) Th ee-channel unmixing asks
Task Da ase Task De ails 2D/3D PSNR Mic oMS-SSIM
C1 C2 C3 C1 C2 C3
XXI CBG-Z18 TM-I 3D 28.1 28.9 29.1 0.920 0.929 0.913
XXII CBG-N18 TM-I 3D 38.4 42.4 35.0 0.977 0.981 0.974
XXIII HHMI-D258bi TM-I 3D 22.5 31.3 24.3 0.840 0.768 0.793
XXIV HT-LIF24 TM-III, 2ms 2D 31.0 32.2 36.3 0.940 0.973 0.944
XXV HT-LIF24 TM-III, 3ms 2D 30.8 32.2 36.1 0.940 0.973 0.948
XXVI HT-LIF24 TM-III, 5ms 2D 32.9 34.3 37.6 0.960 0.983 0.963
XXVII HT-LIF24 TM-III, 20ms 2D 37.0 39.7 41.4 0.984 0.994 0.989
XXVIII HT-LIF24 TM-III, .5s 2D 39.5 41.6 43.1 0.991 0.995 0.995
(c) Fou -channel unmixing asks
Task Da ase 2D/3D PSNR Mic oMS-SSIM
C1 C2 C3 C4 C1 C2 C3 C4
XXIX HT-LIF24 2D 32.1 32.5 34.2 37.6 0.964 0.959 0.983 0.957
XXX Chicago-Sch23 2D 35.6 39.4 38.8 36.2 0.956 0.986 0.980 0.942
Table 1 Quan i a i e pe o mance on wo-, h ee-, and ou -channel spli ing asks. Th oughou his wo k, we
e e o speci ic spli ing asks by hei Task-Id ( i s column). Some o he mic oscopy da ase s we use (column wo), gi e
ise o mul iple spli ing asks, e.g. by selec ing a subse o he luo escen channels o using di e en noise le els o channel
mixing weigh s (see Sec ion 4.1). Such Task De ails a e in abb e ia ed o m gi en in he hi d column along wi h he
aining mode used o he ask. Fo asks wi h Task-Id XXIX and XXX, he aining modes used we e TM-IIIand TM-I,
espec i ely. Fo olume ic asks (labeled 3D in column 2D/3D), we ed a 3D image s ack o Mic oSpli , which in u n also
p edic s 3D ou pu s (pos e io samples). We e alua e all he asks on held-ou es se s and epo Mic oMS-SSIM [8] and
CARE-PSNR (PSNR) me ics o each unmixed channel p edic ion. In Table G.3 we lis all s anda d e o s o he esul s in
he abo e able. No e ha Task XXIII is he mos di icul seman ic unmixing ask o isually di e en cellula s uc u es we
encoun e ed so a , pa icula ly w. . . he quali y o p edic ions o he hi d channel. In Sec ion 2.5 and Supplemen a y
Sec ion B.1.1, we elabo a e on how low SNR and se e al o he p ope ies o he aw da a a e con ibu ing o he simplici y
o di icul y o seman ic unmixing, and how po en ially occu ing p oblems can be mi iga ed.
In cases whe e mul iplexed da a o T aining Mode I a e no a ailable, we p opose T aining Mode II .
He e, we assume ha images o each single s uc u e exis , bu we no longe assume ha all s uc u es ha e
been imaged in each sample. As be o e, he e we also gene a e supe imposed inpu s by summing images
showing di e en luo escen s uc u es. Howe e , unlike be o e, summed-up inpu images no longe o igina e
om he same sample, and any spa ial co ela ions ha migh exis be ween hese cellula s uc u es
a e los . Since a ne wo k ained wi h T aining Mode II canno le e age hese co ela ions, we easoned
ha T aining Mode I should be a leas as pe o man as T aining Mode II . In Sec ion C.2 we es his
hypo hesis and show ha T aining Mode II indeed comes wi h a sligh pe o mance d op in cases whe e
he channels o be unmixed o e ac ionable spa ial co ela ions ha Mic oSpli can le e age.
4
A a ia ion o his app oach, T aining Mode II-b, was used o ob ain he esul s shown in
Sec ion 2.4.2. Ins ead o summing unco ela ed se s o images, we used image da a o supe imposed s uc-
u es. I hose s uc u es mix in some a eas, bu a e isible in isola ion in o he s, we c opped egions
o in e es (ROIs) showing indi idual s uc u es and summed hem andomly in o supe imposed aining
inpu s. This aining mode is bes unde s ood in he con ex o he esul s we p esen in Sec ion 2.4.2.
Finally, in T aining Mode III , we do no c ea e inpu images by summing images o indi idual s uc-
u es bu a he acqui e hem also a he mic oscope. In his mode, as in T aining Mode I , all s uc u es o
in e es mus be indi idually labeled o allow mul iplexed imaging o he equi ed aining a ge s. Addi ion-
ally, we acqui e an addi ional image channel by exci ing all used labels a once and collec ing he en i e y
o he emi ed ligh , hence, di ec ly imaging also he supe imposed inpu di ec ly a he mic oscope. Na u-
ally, T aining Mode III hen uses his channel ins ead o he summa ion o he a ge channels as inpu o
Mic oSpli . The ad an age o his is ha he inpu image is also subjec o ealis ic image noise and ha
he ela i e in ensi y o he di e en s uc u es is ealis ic as well (see esul s o asks wi h IDs IV-XIV
and XXIII-XXIX).
These aining modes p o ide lexibili y in he p epa a ion o aining da a o Mic oSpli , ensu ing
obus pe o mance unde a a ie y o expe imen al condi ions and noise le els. We p o ide an o e iew o
aining modes in Supplemen a y Table G.3.
2.2 Mic oSpli Yields High-Quali y Unmixed S uc u es
To explo e he pe o mance o Mic oSpli in a ious biological samples, imaging modali ies, and aining
modes (see Sec ion 2.1), we collec ed a o al o 10 da a se s and de ined a o al o 30 + 6 seman ic unmixing
asks, as shown in Table 1and Table C.3. A b ie o e iew o he da ase s can be ound in he Online
Me hods (Sec ion 4) and mo e de ails a e gi en in Supplemen a y Sec ion F.
In Figu e 1d- we show quali a i e esul s o h ee o hese asks. A quan i a i e assessmen o he
a ailable g ound u h, called aining a ge s h oughou his manusc ip , can be ound in Table 1. In
he same able, we lis all asks and he achie ed quali y o unmixed channels in e ms o CARE-PSNR
(PSNR) [1] and Mic oMS-SSIM [8], a a ia ion on SSIM op imized o quan i a i e e alua ion o mic oscopy
image da a.
Th oughou all asks, we obse ed a e age PSNR and Mic oMS-SSIM alues o 32.53 and 0.886, espec-
i ely, which o mo e common asks, such as image denoising, would ypically be conside ed high enough
o wa an downs eam p ocessing and analysis. The lowes sco e o all seman ic unmixing expe imen s s ill
shows PSNR/Mic oMS-SSIM alues o 21.6/0.564 (Task III-channel 2 and Task VI-channel 2, espec i ely),
which, depending on he desi ed analysis o be pe o med, is a guably s ill eliable enough o downs eam
analysis.
Howe e , nei he PSNR no Mic oMS-SSIM a e su icien o know how us wo hy he p edic ions o
Mic oSpli a e, since hese me ics can only be calcula ed when g ound u h a ge s a e a ailable. Fo
his eason, Mic oSpli employs a a ia ional aining pa adigm o i s Spli ing Encode -Decode Neu al
Ne wo k, as shown in Figu e 2. The ne wo k a chi ec u e we use is simila o a hie a chical a ia ional
au oencode (HVAE) [10], some imes also e e ed o as a Ladde -VAE [11]. The mos p ominen di e ence
in ou se up is ha Mic oSpli is no an au oencode , since he p edic ions a e no mean o be he same as
he gi en inpu . De ails abou he p ecise ne wo k a chi ec u es and he aining p ocedu e o Mic oSpli
can be ound in Sec ion 4.3, as well as in Sec ion A.1 and in [6,7]. In he nex sec ion, we show how he
a ia ional na u e o Mic oSpli can be exploi ed o es ima e p edic ion e o s caused by ambiguous inpu s
being ed.
2.3 E o Es ima ion, Da a Unce ain y, and Calib a ion
Mic oSpli exploi s he a ia ional na u e o i s unde lying a chi ec u e o enable unce ain y quan i ica-
ion. Since a ia ional ne wo ks a e no simple poin -p edic o s, bu ins ead a e capable o lea ning an
app oxima e pos e io o possible solu ions [1], Mic oSpli can e icien ly sample such solu ions. As o
VAEs [12], mo e likely solu ions will be sampled mo e equen ly, sugges ing ha he analysis o he in e -
sample a iabili y can be a good su oga e o he unce ain y in he inpu da a and, he e o e, also o he
us wo hiness o seman ic unmixing esul s.
We es ed his hypo hesis and show ou indings in Figu e 2b,c. Mo e speci ically, we show an inpu
pa ch and he la e al con ex ha was ed o Mic oSpli , wo pos e io samples, he di e ence be ween
hem as a hea map, and he (app oxima e) minimum mean squa ed e o (MMSE) p edic ion, ob ained
5

Fig. 2 Ne wo k A chi ec u e, Pos e io Sampling, and Calib a ion. (a) Mic oSpli join ly lea ns o denoise inpu
images (unsupe ised, blue shaded a ea) and spli supe imposed s uc u es (supe ised, ed shaded a ea). Fo his, we use
a hie a chical ne wo k a chi ec u e, a ia ional aining (yellow shaded a ea), and use la e al con ex (LC, [6]) o be e
pe o mance. Due o unsupe ised denoising, using an adequa e noise model [7,9], he supe ised a ge images can be subjec
o noise, and he inal p edic ion will s ill be noise- ee. (b) Le mos column shows an inpu pa ch con aining h ee labeled
s uc u es in a single channel a na i e esolu ion (bo om) and wo LC inpu pa ches ha add addi ional spa ial con ex
su ounding he p ima y inpu pa ch ( owa ds he op). Since he inpu is noisy, no all s uc u al de ails in he da a a e
isible. To accoun o his noise-induced da a unce ain y, he a ia ional a chi ec u e we use is capable o sampling plausible
“in e p e a ions” om a lea ned pos e io o possible solu ions. Fo each o he h ee supe imposed image s uc u es (C1-C3)
we show wo such pos e io samples in columns wo and h ee, and hei di e ence as a hea map in column ou . The i h
column shows an app oxima ion o he pos e io minimum mean-squa ed-e o (MMSE), compu ed by pixel-wise a e aging 50
pos e io samples. The las h ee columns show he noisy a ge images (as used du ing aining), an es ima e o he pixel-
wise oo mean squa e e o (RMSE) and he calib a ion plo s o he h ee channels, espec i ely. The calib a ion plo s show
ha he unce ain y es ima ed solely om p edic ions (RMV) co ela e wi h he ue p edic ion e o (RMSE), which means
ha his easy- o-calcula e quan i y (RMV) can be used as a su oga e o an o he wise di icul - o-assess p edic ion e o o
Mic oSpli . (c) Ten pos e io samples o C3, he channel showing he highes es ima ed RMSE alues in (b). No e ha punc a
a low es ima ed RMSE alue loca ions (cyan a ows) emain ela i ely unchanged and closely esemble he s uc u e p esen
in he noisy a ge a ha loca ion, while punc a a high es ima ed RMSE loca ions ( ed a ows) show signi ican a ia ions
be ween samples in (c).
by pixelwise a e aging o 50 pos e io samples. We also show he a ge pa ches used o ain he shown
3-channel spli ing ask (Task XXIII). No e also he e how much lowe he signal- o-noise a io is in he
aining da a han in he p edic ions o a ained Mic oSpli ne wo k. Finally, we show he es ima ed pixel-
wise oo mean squa ed e o (RMSE), compu ed using he 50 pos e io samples ha we also used o
gene a e he p e iously shown MMSE solu ion.
The o e all idea migh be bes con eyed by looking a a conc e e example. Cyan and ed a ows in
Figu e 2b,c poin a image loca ions ha migh show punc a-like s uc u es. No e ha in he 10 pos e io
samples shown in Figu e 2c, he loca ions poin ed a wi h cyan a ows emain ela i ely unchanged and he
p edic ions ma ch he s uc u e p esen in he a ge channel. Howe e , hose loca ions poin ed a wi h ed
a ows change hei appea ance conside ably om one pos e io sample o he nex . This is, in ac , also
e lec ed in he es ima ed RMSE pa ches shown in Figu e 2b, whe e highe es ima ed RMSE alues appea
p ecisely a loca ions whe e he pos e io samples show ele a ed a iabili y and is hence indica ing ha
6
Fig. 3 Segmen a ions on Unmixed Channels a e in line wi h In e -Obse e Va iabili y. We de ine h ee seg-
men a ion asks, each on one unmixed channel p edic ed by Mic oSpli , and compa e he segmen a ions c ea ed by h ee
Bioimage Analys s wi h each o he . (a) Fo he h ee asks a hand, we show o e iew images (le ) and inse s ( igh ) o h ee
wo-channel spli ing asks, wi h he supe imposed aw inpu image in he op ow, he p edic ed channel o be segmen ed in
he middle ow, and a single-channel con ol acquisi ions ia egula mul iplexing ( a ge , see Figu e 1a) in he bo om ow.
(b, c) Fo Task 1 and Task 2, we show segmen a ion esul s ob ained by h ee analys s (A1-A3) ob ained on he Mic oSpli
p edic ions ( op ow), he single-channel con ol acquisi ions ( a ge , middle ow). The igh mos column shows o e lays o he
esul s ob ained by all h ee analys s, wi h consis en ly segmen ed pixels being shown in whi e. The bo om ow i s shows
he o e lay o he segmen a ion esul s o a single analys on p edic ed ( op) s. single channel con ol inpu s (middle), and
in he las column a box plo o all pai wise DICE sco es be ween p edic ed s. a ge channel inpu s o he en i e es se
(5 images o size 1600 ×1600 o Task 1 and size 4096 ×4096 o Task 2). No e ha he in a-obse e a iabili y be ween
p edic ed and a ge segmen a ions ( i s h ee box-plo s) a e in abou he same ange as he in e -obse e a iabili y shown
in he igh mos box-plo .
7
he inpu da a is ambiguous in hese egions. Howe e , whe he a iabili y in pos e io samples ac ually
mani es s i sel in loca ions whe e p edic ions o Mic oSpli a e p one o e o and, con e sely, also only in
hese places, is no ob ious. Pos e io samples could consis en ly p edic s uc u es in places whe e hey
a e no , o consis en ly p edic he absence o s uc u es whe e, in ac , such s uc u es should be. To show
ha a ained Mic oSpli ne wo k does no consis en ly hallucina e he p esence o absence o s uc u es,
we p opose o check how well calib a ed he ne wo k is [7,13–15].
A calib a ed neu al ne wo k is one whose p edic ed p obabili ies o unce ain ies accu a ely e lec
he ue likelihood o ou comes. In he con ex o a eg ession ask like he one execu ed by Mic oSpli ,
calib a ion means ha he p edic ed e o es ima es o he model align wi h he ac ual dis ibu ion o
e o s in he da a. We he e o e es ima ed he ue e o by compu ing he pixel e o be ween he MMSE
p edic ion o Mic oSpli and he g ound u h a ge images we de i ed om he a ailable aining da a,
and plo ed his “ ue e o ” agains he RMSE e o s we desc ibed abo e. As he calib a ion plo s in
Figu e 2b show, he ue e o and RMSE scale oughly linea ly wi h espec o each o he , meaning ha
he es ima ed RMSE, which can be compu ed om pos e io samples only, is a good es ima o o he ue
e o . Hence, calib a ed Mic oSpli ne wo ks o e a eliable and e icien way o es ima e he unce ain y o
he da a and, he e o e, also he magni ude o he e o o i s p edic ions. This p ope y is a conside able
ad an age o e non- a ia ional app oaches and will help u u e use s o (i) il e images o image egions
ha lead o unce ain p edic ions, and (ii) p o ide e idence o hemsel es and o he s ha unmixed images
do no su e om hallucina ions.
2.4 Downs eam P ocessing o Unmixed Da a
Sample p epa a ion and mic oscopy da a acquisi ion a e ypically only he i s s eps in he pu sui o
scien i ic insigh . Once images ha e been acqui ed, hei con en mus be analyzed and quan i ied. In
his wo k, we canno co e he ull b ead h o image da a analyses ha li e scien is s ou inely pe o m
on mic oscopy da a [16]. Howe e , in many analysis pipelines, image segmen a ion is a key s ep since
i de e mines he iden i y, loca ion, shape, and ela i e loca ion o biological en i ies o in e es . In he
ollowing sec ion, we ha e quan i ied he segmen a ion pe o mance on Mic oSpli p edic ions compa ed o
he same analysis ca ied ou on adi ionally mul iplexed image da a.
2.4.1 Segmen a ion o Unmixed Image Da a is o High Quali y
We show ha a ypical segmen a ion ask, as i is conduc ed in biological esea ch labo a o ies on a daily
basis, leads o compa able quali y esul s when conduc ed on egula mul iplexed mic oscopy da a o on
unmixed images. To his end, we compa ed he segmen a ion esul s o h ee bioimage analys s who we e
ins uc ed o in e ac i ely ain a pixel-classi ie un il hey each bes -possible esul s. We show he esul s
in Figu e 3and Figu e S7, and explain he de ails o his expe imen below.
Fi s , each analys segmen ed he single-channel mic oscopy da a acqui ed by egula mul iplexed luo-
escen mic oscopy (Ta ge ) and he wo-channel seman ic unmixing p edic ions o Mic oSpli (P edic ion)
wi hou being in o med abou he na u e o he da a. We hen compa ed he consis ency be ween he seg-
men a ions o each analys on he wo se s o da a and he consis ency o he segmen a ions be ween he
analys s. We obse ed ha he a iabili y be ween segmen a ions on a ge and unmixed images lies wi hin
he ange o he in e -obse e a iabili y be ween he h ee analys s. We quali a i ely show he segmen-
a ion esul s on wo expe imen s (Tasks) in Figu e 3bc and plo he measu ed in a-obse e a iabili ies
(A1,A2,A3) and he in e -obse e a iabili y.
Hence, in all expe imen s we conduc ed, he quali y o segmen a ions emained unchanged when using
unmixed image da a compa ed o mul iplexed images. Bu mo e in e es ingly, imaging wo s uc u es in a
single channel can ee up pho on budge , which hen becomes a ailable o imaging a a highe ame a e,
imaging a a highe signal- o-noise a io, o imaging mo e labeled s uc u es in addi ional image channels.
2.4.2 Remo ing Unwan ed Imaging A i ac s
Na u ally, Mic oSpli p edic ions a e jus images and any subsequen analysis can be pe o med wi h hem.
As a second downs eam p ocessing example, we p esen an inno a i e way o emo e imaging a i ac s.
Mo e speci ically, we imaged he pos -mi o ic neu onal ma ke CTIP2 in sliced hPSC-de i ed o eb ain
o ganoids using a seconda y an ibody conjuga ed o a 555 dye. The esul ing image da a no only had
CTIP2+ nuclea s aining (wan ed signal), bu also punc a ha accumula ed because o non-speci ic signal
(i.e. undesi ed a i ac s).
8
Fig. 4 Remo ing Image A i ac s wi h Mic oSpli . We emo e unwan ed punc a p esen in aw mic oscopy da a using
ou image unmixing app oach by (i) selec ing a se o c opped egions o in e es (ROI) in he aw da a ha do no con ain
he unwan ed s uc u es (punc a, in his case), and o he se o ROIs ha do only con ain said punc a wi hou he luo escen
signal o in e es , ollowed by (ii) c ea ing supe imposed inpu images by mixing (summing) andomly sampled ROIs o he
wo collec ed se s. These mixed images, oge he wi h he o iginal pai s o ROIs, a e now he inpu and a ge inpu used o
ain a Mic oSpli ne wo k. (a) An example o a aw inpu image (le ), showing labeled nuclei and unwan ed punc a, and
he punc a- ee p edic ion by Mic oSpli . Bo h images show in ensi y his og ams in cyan and ed, espec i ely. The dashed
g ay plo in he p edic ed image co esponds o he cyan plo o he inpu . The y-axes a e shown on a loga i hmic scale o
be e emphasize he punc a in ensi ies Mic oSpli success ully emo ed. (b) Fo be e isual compa ison, we show inpu and
p edic ion inse s om (a) in he op wo ows, espec i ely, and he di e ence images (inpu - p edic ion) in he bo om ow
( ed, in ensi ies emo ed; g ay in ensi ies unchanged; blue, in ensi ies inc eased).
Ideally, we could use Mic oSpli o unmix he labeled nuclei (desi ed image con en ) and he undesi ed
punc a (i.e. imaging a i ac s). Howe e , o his we would need aining a ge s ha con ain only nuclei
and o he s ha con ain only punc a, a equi emen ha he expe imen al se up o ou colleagues does no
pe mi . In Sec ion 2.1, we in oduce T aining Mode II-b, he manual selec ion o image egions ha show
only he a i ac s (punc a) o only he desi ed s uc u es ( he signal, in his example, nuclei). The selec ed
image egions a e hen combined, jus as in any T aining Mode II expe imen , and used o ain a Mic oSpli
ne wo k. Since c ops wi hou punc a o c ops ha show only punc a a e o limi ed size wi h espec o ull
mic oscopy images ( he smalles c ops we collec ed we e 100 ×100 pixels), la e al con ex ualiza ion (LC;
see Figu e 2a) could no be used and was he e o e disabled du ing aining. In o al, we ha e manually
c opped 82 con en egions and 48 punc a egions om he da a. This led o a o al o 53.7Mpixels o
aining and ook an analys abou eigh hou s o ocused wo k.
A e aining had con e ged, we e alua ed he model on p e iously held ou ull-size mic oscopy images.
A ep esen a i e esul is shown in Figu e 4a,b. The inse s in Figu e 4b allow a mo e de ailed compa ison
9
Re e ences
[1] Sh o , H., Tes a, I., Jug, F., Manley, S.: Li e-cell imaging powe ed by compu a ion. Na . Re . Mol. Cell Biol.
(2024)
[2] Weige , M., Roye , L., Jug, F., Mye s, G.: Iso opic econs uc ion o 3d luo escence mic oscopy images using
con olu ional neu al ne wo ks. In: Desco eaux, M., Maie -Hein, L., F anz, A., Jannin, P., Collins, D.L., Duch-
esne, S. (eds.) Medical Image Compu ing and Compu e -Assis ed In e en ion - MICCAI 2017, pp. 126–134.
Sp inge , Cham (2017)
[3] Weige , M., Schmid , U., Boo he, T., M¨ulle , A., Dib o , A., Jain, A., Wilhelm, B., Schmid , D., B oaddus,
C., Culley, S., Rocha-Ma ins, M., Sego ia-Mi anda, F., No den, C., Hen iques, R., Ze ial, M., Solimena, M.,
Rink, J., Tomancak, P., Roye , L., Jug, F., Mye s, E.W.: Con en -awa e image es o a ion: pushing he limi s
o luo escence mic oscopy. Na u e Publishing G oup 15(12), 1090–1097 (2018)
[4] K ull, A., Buchholz, T.-O., Jug, F.: Noise2Void - lea ning denoising om single noisy images, pp. 2129–2137
(2019)
[5] P akash, M., K ull, A., Jug, F.: Di Noising: Di e si y denoising wi h ully con olu ional a ia ional au oen-
code s. ICLR 2020 (2020)
[6] Ashesh, K ull, A., San e, M.D., Pasqualini, F.S., Jug, F.: µSpli : image decomposi ion o luo escence
mic oscopy. (2023)
[7] Ashesh, A., Jug, F.: denoispli : A me hod o join mic oscopy image spli ing and unsupe ised denoising. In:
Leona dis, A., Ricci, E., Ro h, S., Russako sky, O., Sa le , T., Va ol, G. (eds.) Compu e Vision – ECCV 2024,
pp. 222–237. Sp inge , Cham (2025)
[8] Ashesh, A., Deschamps, J., Jug, F.: Mic oSSIM: Imp o ed S uc u al Simila i y o Compa ing Mic oscopy
Da a. In: BIC Wo kshop, ECCV, Milan (2024). h ps://doi.o g/10.48550/a Xi .2408.08747
[9] P akash, M., Delb acio, M., Milan a , P., Jug, F.: In e p e able unsupe ised di e si y denoising and a e ac
emo al. (2021)
[10] Vahda , A., Kau z, J.: NVAE: A deep hie a chical a ia ional au oencode . In: Neu al In o ma ion P ocessing
Sys ems (Neu IPS) (2020)
[11] Sønde by, C.K., Raiko, T., Maaløe, L., Sønde by, S.K., Win he , O.: Ladde a ia ional au oencode s. In:
P oceedings o he 30 h In e na ional Con e ence on Neu al In o ma ion P ocessing Sys ems. NIPS’16, pp.
3745–3753. Cu an Associa es Inc., Red Hook, NY, USA (2016)
[12] Kingma, D.P., Welling, M.: An in oduc ion o a ia ional au oencode s (2019) a Xi :1906.02691 [cs.LG]
[13] DeG oo , M.H., Fienbe g, S.E.: The compa ison and e alua ion o o ecas e s. Jou nal o he Royal S a is ical
Socie y. Se ies D (The S a is ician) 32(1), 12–22 (1983). Accessed 2025-01-26
[14] Niculescu-Mizil, A., Ca uana, R.: P edic ing good p obabili ies wi h supe ised lea ning. In: P oceed-
ings o he 22nd In e na ional Con e ence on Machine Lea ning. ICML ’05, pp. 625–632. Associa-
ion o Compu ing Machine y, New Yo k, NY, USA (2005). h ps://doi.o g/10.1145/1102351.1102430 .
h ps://doi.o g/10.1145/1102351.1102430
[15] Le i, D., Gispan, L., Giladi, N., Fe aya, E.: E alua ing and calib a ing unce ain y p edic ion in eg ession
asks. Senso s 22(15) (2022)
[16] Sch oede , A.B., Dobson, E.T.A., Rueden, C.T., Tomancak, P., Jug, F., Elicei i, K.W.: The ImageJ ecosys em:
Open-sou ce so wa e o image isualiza ion, p ocessing, and analysis. P o ein Sci. 30(1), 234–249 (2021)
[17] Blau, Y., Michaeli, T.: The pe cep ion-dis o ion adeo . In: 2018 IEEE Con e ence on Com-
pu e Vision and Pa e n Recogni ion, CVPR 2018, Sal Lake Ci y, UT, USA, June 18-22, 2018,
pp. 6228–6237. Compu e Vision Founda ion / IEEE Compu e Socie y, ??? (2018). h ps://doi.
o g/10.1109/CVPR.2018.00652 .h p://openaccess. hec .com/con en c p 2018/h ml/Blau The Pe cep ion-
Dis o ion T adeo CVPR 2018 pape .h ml
16

[18] Sh o , H., Tes a, I., Jug, F., Manley, S.: Li e-cell imaging powe ed by compu a ion. Na u e Re iews Molecula
Cell Biology 25(6), 443–463 (2024) h ps://doi.o g/10.1038/s41580-024-00702-6 . Accessed 2025-11-18
[19] Zimme mann, T.: Spec al imaging and linea unmixing in ligh mic oscopy. Ad . Biochem. Eng. Bio echnol.
95, 245–265 (2005)
[20] Seo, J., Sim, Y., Kim, J., Kim, H., Cho, I., Nam, H., Yoon, Y.-G., Chang, J.-B.: PICASSO allows ul a-
mul iplexed luo escence imaging o spa ially o e lapping p o eins wi hou e e ence spec a measu emen s. Na .
Commun. 13(1), 1–17 (2022)
[21] Noga e, D.D., Ha ley, M., Deschamps, J., Ellenbe g, J., Jug, F.: Using AI in bioimage analysis o ele a e he
a e o scien i ic disco e y as a communi y. Na . Me hods 20(7), 973–975 (2023)
[22] Deschamps, J., Dalle Noga e, D., Jug, F.: Be e esea ch so wa e ools o ele a e he a e o scien i ic disco e y
o why we need o in es in esea ch so wa e enginee ing. F on Bioin o m 3, 1255159 (2023)
[23] Lapo e, M.H., Klena, N., Hamel, V., Guicha d, P.: Visualizing he na i e cellula o ganiza ion by coupling
c yo ixa ion wi h expansion mic oscopy (C yo-ExM). Na u e Me hods 19(2), 216–222 (2022) h ps://doi.o g/
10.1038/s41592-021-01356-4 . Accessed 2025-02-02
[24] Gong, D., Cai, C., S ahile i z, E., Chen, J., Sche e , N.F.: Easily scalable mul i-colo DMD-based s uc u ed
illumina ion mic oscopy. Op . Le . 49(1), 77–80 (2024)
[25] Wang, Z., Bo ik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quali y assessmen : om e o isibili y o s uc-
u al simila i y. IEEE T ansac ions on Image P ocessing 13(4), 600–612 (2004) h ps://doi.o g/10.1109/TIP.
2003.819861
[26] Wang, Z., Bo ik, A.C.: A uni e sal image quali y index. IEEE Signal P ocessing Le e s 9(3), 81–84 (2002)
h ps://doi.o g/10.1109/97.995823
[27] Wang, Z., Simoncelli, E.P., Bo ik, A.C.: Mul iscale s uc u al simila i y o image quali y assessmen . In: The
Th i y-Se en h Asiloma Con e ence on Signals, Sys ems and Compu e s, 2003, ol. 2, pp. 1398–14022 (2003).
h ps://doi.o g/10.1109/ACSSC.2003.1292216
[28] Di San e, M., Pezzo i, M., Zimme mann, J., En ico, A., Deschamps, J., Balmas, E., Becca, S.,
Reali, A., Be e o, A., Jug, F., Pasqualini, F.S.: Calipe s: Cell cycle-awa e li e imaging o pheno-
yping expe imen s and egene a ion s udies. bioRxi (2024) h ps://doi.o g/10.1101/2024.12.19.629259
h ps://www.bio xi .o g/con en /ea ly/2024/12/22/2024.12.19.629259. ull.pd
[29] Walsh, R.M., Luongo, R., Giacomelli, E., Cice i, G., Ri enhouse, C., Ve illo, A., Galimbe i, M., Bocchi, V.D.,
Wu, Y., Xu, N., Mosole, S., Mulle , J., Vezzoli, E., Jung e do ben, J., Zhou, T., Ba ke , R.A., Ca aneo, E.,
S ude , L., Baggiolini, A.: Gene a ion o human ce eb al o ganoids wi h a s uc u ed ou e sub en icula zone.
Cell Repo s 43(4), 114031 (2024) h ps://doi.o g/10.1016/j.cel ep.2024.114031 . Accessed 2025-07-21
17
A Model A chi ec u e and T aining
In Mic oSpli , we ha e combined he bene i s o µSpli [6] and denoiSpli [7], cas hose ideas in a common
lea ning amewo k, enabled di ec aining and p edic ion on olume ic da a, and ha e ex ensi ely es ed
and e alua ed i s pe o mance on a wide ange o da ase s and seman ic unmixing asks. We ha e also made
openly a ailable all aining and p edic ion code and all da a we used. We do his o os e apid adop ion
o Mic oSpli by he scien i ic communi y and o allow o he s o imp o e ou app oach and compa e ou
esul s wi h ela i e ease. In his sec ion, we will discuss in de ail he aspec s o he abo e-men ioned
wo ks ha we e in eg a ed in o Mic oSpli . Subsequen ly, we also desc ibe ou loss unc ion and impo an
aining hype pa ame e s.
F om µSpli , we inhe i he abili y o e icien ly inco po a e spa ial con ex using addi ional inpu s,
called La e ally Con ex ualiza ion (LC) inpu s [6]. He e we eed, nex o he p ima y inpu , o which he
p edic ion will be made, addi ional LC inpu s ha help he ne wo k o be e unde s and he image con ex
om which he p ima y inpu is aken (see Figu e 2a and S1). These successi e LC inpu s a e la ge and
la ge pa ches cen e ed on he p ima y inpu pa ch, bu downscaled o he same pixel dimensions as he
p ima y inpu i sel . Hence, LC inpu s cap u e he spa ial con ex a ound he p ima y inpu pa ch bu do
so a lowe esolu ions o ensu e e icien lea ning and p edic ions in a easonably sized o e all ne wo k.
The ne wo k a chi ec u es we p oposed come in la o s ha ade compu a ional complexi y and GPU
consump ion wi h he bes possible p edic ion quali y. I s mos GPU-e icien a ian , Lean-LC, can ain
on a single GPU using less han 5GB o GPU memo y. I mo e esou ces a e a ailable, i is ad isable o op
o se ups such as Deep-LC, which show be e p edic i e pe o mance a inc eased compu a ional cos .
In Figu e S1, we ha e b ie ly desc ibed he h ee µSpli a ian s. The whi e egions co espond o
ea u es o igina ing om he p ima y inpu pa ch. As in U-Ne a chi ec u es, spa ial esolu ion hal es
a each successi e hie a chy le el h ough pooling ope a ions, causing hese whi e a eas o p og essi ely
diminish in size. In µSpli (and hence also in Mic oSpli ), he pooled embeddings unde go ze o-padding
be o e being conca ena ed wi h ea u e maps om la e al con ex ualiza ion (LC) inpu s. These LC ea u es
a e p ocessed h ough dedica ed ‘Inpu b anch’ sub-ne wo ks consis ing o con olu ional laye s wi h non-
linea ac i a ions, d opou , and no maliza ion componen s. ‘Inpu b anch’ does no ha e pooling ope a ions,
and so hei ea u e maps, a each hie a chy le el, main ain spa ial dimensions iden ical o he p ima y-
inpu ( ep esen ed by g ay egions). This p ese a ion enables me ged embeddings om bo h pa hways o
e ain he o iginal spa ial dimensions h oughou he ne wo k hie a chy (g ay squa es). This is he co e idea
o LC Mic oSpli has inhe i ed om µSpli . In he cap ion o Figu e S1 we p o ide mo e de ails ega ding
he di e ences in he h ee a ian s o µSpli . Please e e o [6] o mo e de ails.
F om denoiSpli , we inhe i he abili y o join ly pe o m unsupe ised denoising, using sui able Noise
Models. This also enables Mic oSpli o sample di e se p edic ions om a lea ned app oxima e pos e io
ha cap u es a no ion o he da a unce ain y, as demons a ed by ou ained ne wo ks being calib a ed
(see Sec ion 2.3). While also µSpli is a a ia ional app oach ha is in heo y capable o gene a ing mul iple
p edic ions om i s pos e io , we ound ha denoiSpli , a guably due o i s di e en KL-loss o mula ion,
p oduces a highe di e si y ha is be e in line wi h he unce ain y in he da a. In Figu e S1, we p esen
he a chi ec u e o Mic oSpli , which has LC inpu s and Noise models, all in eg a ed in o a single se up.
As men ioned abo e, we also enabled Mic oSpli o ope a e di ec ly on olume ic image da a, a
possibili y ha was absen in bo h µSpli and denoiSpli .
A.1 Loss Func ion used o ain Mic oSpli
The loss unc ion o Mic oSpli is he weigh ed a e age be ween he µSpli loss and denoiSpli loss, ha is,
lossMic oSpli =w∗lossdenoiSpli + (1 −w)∗lossµSpli (2)
Unless explici ly speci ied, w= 0.9 is used in all expe imen s we conduc ed. This simple design also gi es
us he abili y o swi ch o pu e µSpli o denoiSpli mode by simply se ing w o 0 o 1, espec i ely.
To inco po a e LC inpu s in o he denoiSpli se up, we obse ed he need o modi y he KL loss o -
mula ion used in lossdenoiSpli . In denoiSpli , pixel-wise KL di e gence is compu ed a e e y hie a chy le el.
Le KLideno e he pixel-wise KL di e gence enso a he i h hie a chy le el. KL-loss componen o his
hie a chy le el, kliis de ined as
kli=α·X
j,h,w
KLi[j, h, w].(3)
18
Fig. S1 Ne wo k a chi ec u es employed by Mic oSpli . This igu e p o ides a de ailed b eakdown o he componen s
shown in Figu e 2a. Speci ically, i illus a es he encode -decode s uc u e o ou a chi ec u e, which is adap ed om ou
p io wo k, µSpli [6]. Since his diag am ocuses on he ans o ma ion o inpu da a in o p edic ions, i does no include he
loss e ms (such as he KL-di e gence loss and he noise-model-based likelihood loss). As in [6], we imp o e he p edic i e
capaci y o Mic oSpli by eeding no only an image pa ch (p ima y inpu ), bu also addi ional image con ex . To his end, we
in oduced la e al con ex ualiza ion (LC). These LC inpu s cap u e a la ge po ion o he inpu da a bu a inc easingly lowe
pixel- esolu ion. This lead o h ee me a-a chi ec u es, Lean-LC, Regula -LC, and Deep-LC, each wi h inc easingly highe
compu a ional and GPU memo y demands bu also leading o inc easingly highe unmixing pe o mance. The g ay egion
shown in he embeddings in all h ee me a-a chi ec u es ep esen s he ex a spa ial size in he embeddings coming solely due
o LC inpu s. In Lean-LC, embeddings o di e en hie a chy le els in he encode bene i om LC inpu s, bu he decode
does no . In Regula -LC, a mo e GPU-consuming a ia ion, bo h he encode ’s and he decode ’s embeddings use LC (i.e.
show g ay egions in he igu e). Finally, since in Regula -LC he spa ial dimensions o he embedding laye s do no dec ease,
i enabled us o s ack some addi ional hie a chy le els on op o he p e iously used ones. We labeled he esul ing a chi ec u e
as Deep-LC, he mos pe o man bu also mos esou ce hea y a ia ion. In o de o ge bes esul s, we ha e used Deep-LC
whene e possible, bu i is impo an o no e ha wi h jus wo simple hype -pa ame e swi ches, one can ins an ly make use
o Regula -LC o Lean-LC and ain on cheape and olde consume GPUs.
19
Wi h LC inpu s, he spa ial dimensions o he la en space enso s and he e o e KLido no dec ease
and he summing ope a ion in his o mula ion leads o a highe alue, owing o he la ge numbe o
summands (which a e all non-nega i e). This gi es unnecessa ily high weigh o he KL loss wi h espec
o he likelihood loss componen . We obse ed ha his deg ades he pe o mance. To handle his, we
cen e -c opped KLi o he shape hey would assume i he e we e no LC inpu s. Le KLc opped
ideno e he
app op ia ely cen e -c opped e sion o KLi. Ou modi ied KL loss o denoiSpli becomes
kli=α·X
j,h,w
KLc opped
i[j, h, w].(4)
We encoun e ed a e y simila issue when wo king wi h olume ic da a. In ha case, pixel-wise KL-
di e gence is a 4-dimensional enso C×Z×Hi×Wi. As we wo k wi h la ge and la ge Z, he summa ion
in Equa ion 3would inc ease since he summa ion would be on all 4 dimensions. This again leads o gi ing
mo e weigh o he KL loss componen agains he likelihood loss, hus ende ing he pe o mance in e io .
No e ha his a ec will become mo e se e e when we inc ease he numbe o Z ames in he inpu . In
o he wo ds, adding mo e in o ma ion in he inpu was no bene icial. To handle his, we sepa a ely ook
ca e o he ex a Zdimension by aking he a e age along his dimension. The esul an 3D enso is hen
passed o Equa ion 4 o compu e he KL loss componen . No e ha he e is an addi ional dimension o
ba ch size which we ha e no men ioned in he abo e explana ion. Tha is because KL-loss is compu ed
sepa a ely o e e y elemen in he ba ch.
A.2 Hype -pa ame e s used du ing T aining
We use PyTo ch package o c ea ing ou aining and e alua ion pipelines. We use a ba ch_size o
32, max_epoch o 400 and lea ning_ a e o 0.001. We use Adamax op imize and ReduceLROnPla eau
as he lea ning a e schedule wi h l _schedule _pa ience se o 150. Du ing aining, we use 16-bi
p ecision. We use 2 LC inpu s in ou Deep-LC con igu a ion (mul iscale_low es_coun = 3). Please e e
o ou code (h ps://gi hub.com/CAREamics/Mic oSpli - ep oducibili y) o mo e de ails. We wan o s a e
ha all expe imen s we e done using code hos ed a h ps://gi hub.com/juglab/Mic oSpli . Howe e , we
ha e de eloped h ps://gi hub.com/CAREamics/Mic oSpli - ep oducibili y wi h he objec i e o p o iding
use - iendly code wi h easie adap abili y o cus om da ase s.
B Analyzing Fac o s ha A ec P edic i e Pe o mance
B.1 Pixel-noise (pixel-wise independen noise)
F om ou di e en expe imen s, we ha e ound ha he e a e wo p ominen ac o s ha a ec he pe o -
mance on seman ic unmixing asks. The i s ac o is he amoun o pixel-independen noise ha is p esen
in he supe imposed inpu images and a ge images. Mo e noise means in e io pe o mance. To quan i y
he e ec o noise, we imaged ou HT-LIF24 da ase wi h exposu e du a ions o 2ms, 3ms, 5ms, 20ms, and
500ms. We imaged i such ha he unde lying con en in hese sub-da ase s is iden ical. Tha is, o e e y
ame in he 2ms acquisi ion, we ha e he co esponding highe SNR ames in 3ms, 5ms, 20ms and 500ms
acquisi ions. We ained a h ee-channel seman ic unmixing ask sepa a ely o each exposu e du a ion sub-
da ase . We made p edic ions on he held-ou es se inpu ames om hei espec i e exposu e du a ion
sub-da ase s. We e alua e he p edic ion agains he a ge channels p esen in he 500ms sub-da ase .
Al hough he esul s (Tasks XXIV-XXIX) in Table 1show he expec ed end (seman ic unmixing
quali y dec eases when using lowe SNR aining da a), e en he sho es exposu e ime o 2ms s ill lead o
unmixed p edic ions ha a e i o downs eam p ocessing and analysis (in all cases we measu ed a PSNR
>30.8 and Mic oMS-SSIM >0.94). To explain he pe o mance d op, supplemen a y igu e S60 shows ha
educed PSNR p ima ily esul s om he loss o high- equency de ails.
B.1.1 Lessons lea ned om he di icul Task XXIII (on HHMI-D25 da a)
Ou o all asks men ioned in he Table 1, he ask XXIII which uses HHMI-D258bi da ase has conside -
able pe o mance issues, especially in he hi d channel (see Supplemen a y Figu e S57). In he nex ew
pa ag aphs we will desc ibe ou app oach o in es iga ing his issue and p esen a wo king solu ion. We do
his o p o ide an example o how use s o Mic oSpli can imp o e solu ions ha migh ini ially no lead
up o he equi ed seman ic unmixing quali y.
20
Signal- o-noise (SNR) is one o he ac o s which plays a ole in almos all deep-lea ning based me hods
and denoising app oaches, and seman ic unmixing is no excep ion. To in es iga e he ole o SNR o ask
XXIII, we i s denoised he aw da a o HHMI-D258bi using Noise2Void [4], and hen ained Mic oSpli
using hose denoised images, calling his aining ask ‘Task XXXI’. Compa ing he esul s o asks XXIII
and XXXI, see also Table C.3, i becomes appa en ha p io denoising has a a he s ong posi i e e ec
on he quali y o he achie ed seman ic unmixing esul s (>7db PSNR imp o emen ), sugges ing ha he
low SNR in he o iginal da a migh indeed ha e caused he bad pe o mance.
Since HHMI-D258bi is s o ed in unsigned in 8, meaning ha pixel alues a e in [0,255], he e a e ac ually
only a ew dis inc in ege alues p esen ing mos o he da a. Denoising his da a, besides inc easing he
SNR, also inc eases he numbe o unique pixel alues. We hypo hesized ha in addi ion o SNR, an o e ly
disc ee na u e o pixel alues migh also be de imen al o seman ic unmixing. To es his hypo hesis,
we imaged he HHMI-D2516bi da a subse , whe e he unsigned in 16 o ma was used o s o e he da a,
inc easing he pixel in ensi y ange o [0,65535]. Doing so, we ensu ed ha he SNR ( a io o a e age
o eg ound alue o a e age backg ound alue) is as simila as possible o bo h, he 8 and 16 bi e sions
o he HHMI-D25 da a. Using he 16bi da a did indeed imp o e he quali y o p edic ions, also o he
p oblema ic hi d channel, as can be seen in Supplemen a y Figu es S58 and S62. The quan i a i e me ics
in Table C.3, howe e , do no cap u e he imp o emen we can pe cei e by compa ing hose igu es. To ully
alida e ou hypo hesis ega ding SNR and o e ly disc e e pixel in ensi ies, we imaged ano he da a subse
o HHMI-D25, namely HHMI-D2516bi ,0.25, which no only uses unsigned uin 16 o ma bu also bins ou
pixels in o one ( he eby inc easing SNR on he cos o spa ial esolu ion). On his da a subse we de ined
Task XXXVI, and indeed obse e a much imp o ed seman ic unmixing pe o mance (see Table C.3 and
Figu e S59).
We also expe imen ed wi h syn he ic Gaussian and Poisson noise wi h HHMI-D25 da ase e sions. The
mo i a ion was o s a om a wo king se up, make he aining and e alua ion da ase noisy and inspec
he pe o mance deg ada ion. Fo his pu pose we picked HHMI-D2516bi and HHMI-D258bi ,denoised. We
added Gaussian noise (σ) and Poisson noise (λ). Gi en an image x, i s noisy e sion can be exp essed
Poi(x/λ)·λ+ϵ, whe e ϵ∼N(0, σ) and Poi() ep esen s he Poisson dis ibu ion wi h pa ame e λ. As
i was o be expec ed, he pe o mance deg ades wi h noise (see Tasks XXXII, XXXIV, and XXXV in
Supplemen a y Figu es S62, and S61, and a quan i a i e compa ison in Supplemen a y Table C.3.
B.1.2 Ou -o -dis ibu ion SNR
. We also use he se o models ained on di e en HT-LIF24 sub-da ase s o unde s and how he pe o -
mance deg ades wi h ou -o -dis ibu ion inpu s. Fo his, we e alua e he pe o mance o Mic oSpli ained
on one exposu e du a ion on he supe imposed inpu images coming om a di e en exposu e du a ion. We
p esen he esul s in Figu e S5. The di e en cu es ep esen indi idual Mic oSpli models ained on one
speci ic exposu e du a ion sub-da ase as speci ied in he legend. On he x-axis, we ha e di e en e alua ion
sub-da ase s, e e ed o by hei exposu e du a ion. F om CARE-PSNR and Mic oMS-SSIM plo s, one
can obse e ha pe o mance imp o es as one inc eases he exposu e du a ion. Addi ionally, upon obse -
ing pe o mance on 2ms and 500ms sub-da ase s, one can see ha in mos cases, he la ge he di e ence
be ween he exposu e du a ion o he aining sub-da ase and e alua ion sub-da ase , he highe he pe -
o mance d op. Fo ins ance, i we look a CARE-PSNR plo , he wo wo s -pe o ming Mic oSpli models
on 2ms acquisi ion we e ained on 20ms and 500ms. And he wo wo s -pe o ming Mic oSpli models on
500ms acquisi ion we e ained on 2ms and 3ms. On a di e en no e, one can obse e much less a ia ion
in SSIM and MS-SSIM plo s. We discuss his aspec in Sec ion E.
B.2 Spa ial Co ela ion
The s uc u es p esen in he cell a e spa ially co ela ed. Fo example, nuclei a e ypically in he cen al
egions o cells, while he cell bounda y is, by de ini ion, on he bounda y o a cell. The knowledge abou
he cell su ace, he e o e, can ell some hing abou whe e he nucleus should o should no be p esen . We
wan ed o unde s and how impo an his spa ial co ela ion is o ou seman ic unmixing ask.
Fo his, we wo ked wi h he Pa ia-P23 da ase whe e we modi ied he inpu pa ch p ocess du ing
aining. The de aul me hod is o pick a andom loca ion in a ame, ex ac pa ches o bo h s uc u es
( a ge channels) om ha loca ion, and sum hem o c ea e supe imposed inpu (i.e.T aining Mode I).
Using T aining Mode I , he spa ial co ela ion be ween he imaged s uc u es is na u ally main ained. To
dis up his, we conduc ed expe imen s wi h he ollowing al e a ions.
21

In he i s al e a ion, we kep T aining Mode I o 50% o all aining pa ches, bu c ea ed he o he 50%
o aining pa ches by picking wo di e en andom loca ions and adding hem oge he o c ea e an inpu
pa ch (i.e.T aining Mode II ). Hence, we main ain sound spa ial co ela ions be ween he s uc u es o be
unmixed in hal he aining da a. In he second al e a ion, we c ea e all aining da a pa ches acco ding o
T aining Mode II , hus elimina ing all spa ial co ela ions be ween he s uc u es ha should be unmixed
and o cing he ained ne wo k o ely ully on he s uc u al appea ance o he s uc u es only.
We epo he me ics CARE-PSNR and Mic oMS-SSIM in Table ST8 which shows ha he absence o
spa ial co ela ion indeed esul s in a d op in pe o mance by 0.3−0.5 dB PSNR.
B.3 Simila i y o S uc u es o be Unmixed
Since ou me hod elies hea ily on he spa ial appea ance o he s uc u es o be unmixed, we wonde ed
how dissimila wo s uc u es ha e o be o lead o good seman ic unmixing esul s. Hence, i is bes o
image s uc u es in single channels ha a e as dissimila as possible.
In µSpli [6], we p oposed a syn he ic seman ic unmixing da ase based on combina ions o sinusoidal
cu es ha equi ed longe - ange s uc u e in eg a ion. Al hough he s uc u es we e e y simple, he
ne wo k wi hou using LC was unable o spli he inpu and ended up only p edic ing he inpu o bo h
a ge channels equally.
Howe e , o be e e alua e o wha ex en a ne wo k can sepa a e s uc u es ha ha e a simila
appea ance, we designed he ollowing expe imen s. Using he mic o ubule channel o he HT-LIF24 da ase ,
we c ea ed a 2-channel spli ing ask by mixing pa ches om he mic o ubule channel wi h o he pa ches
om he exac same channel. We easoned ha he ne wo k will no be able o spli s uc u es ha a e
li e ally aken om he same se o da a (see bo om ow in Figu e S2). To gi e he ne wo k a chance,
we hen s a ed o al e one o he wo supe imposed copies by scaling he da a (uppe mos ou ows in
Figu e S2). This leads o he supe posi ion o da a ha is s uc u ally s ill e y simila , bu one copy will
ha e a sligh ly la ge appea ance.
In Figu es S2 and S3 we show he esul s o hese expe imen s, conduc ed wi h a ange o di e en
ela i e scaling be ween he wo copies o he mic o ubule da a. The esul s clea ly show ha he ne wo k
is capable o seman ic unmixing he s uc u es in all cases, e en i he scaling ac o was as low as 1.125.
No e also ha i akes he ne wo k inc easingly longe be o e he spli ing pe o mance s a s li ing o and
eaching i s peak, as can be in e ed om he in lec ion poin and con e gence beha io seen in he PSNR
s. aining s eps plo in Figu e S3.
B.4 Unequal Channel In ensi ies
Ano he ac o ha plays a key ole in he pe o mance o Mic oSpli is he skewness in in ensi y be ween
di e en a ge channels in he supe imposed inpu image. This e ec can be caused by di e ging luo opho e
densi ies o di e en s uc u es, by luo opho e bleaching, o by using inadequa e lase lines and/o lase
powe se ings o some luo opho es. Any o his can lead o acquisi ions whe e one o mo e o he s uc u es
a e only weakly p esen in he supe imposed inpu . We hypo hesized ha p edic ions o he b igh e
(dominan ) s uc u e should s ill be o good quali y bu ha he quali y o he dim s uc u e migh be
wo se.
To es his hypo hesis, we wo ked wi h he 2-channel seman ic unmixing asks om he Pa ia-P24
da ase , whe e we a ied he lase powe o he wo channels we imaged om a balanced se ing (50:50) o
inc easingly mo e skewed se ings (i.e. 66:33 and 84:16). No e ha he o al lase powe was kep he same.
We acqui ed images a hese h ee le els o skewness a h ee di e en o al lase powe s and esul ing signal-
o-noise a ios, gi ing us a o al o 9 se s o image acquisi ions. We show he esul s o all seman ic unmixing
expe imen s in Table ST2, whe e ‘Skew’ deno es he asymme y in he powe dis ibu ion (‘Balanced’,
50:50; ‘Mid’, 66:33; ‘High’, 84:16).
We ound ha , ac oss h ee o al lase powe le els, he pe o mance o he b igh i s channel gene ally
inc eases as we go om Balanced o High skew. The second dim channel shows he in e se beha io . No e
ha since he samples we imaged a di e en imaging se ings we e no he same, each expe imen is based
on a di e en body o aining and e alua ion da a. This causes addi ional luc ua ions in he pe o mance
me ics we epo and makes he p esen ed esul s o be non-mono onic. I all 9 imaging sessions had
cap u ed he same samples, we would expec he esul s o show a mono onic end.
22
Unequal Channel In ensi ies - Two Ex eme Examples
Fo a i s example, we wo ked wi h he nucleus and ubulin channels o he Chicago-Sch23 da ase . As
always, we c ea e he supe imposed inpu by summing he wo channels. He e, he nucleus is e y weak
( he channels being e y skewed in hei ela i e in ensi y), so much so ha nuclei a e de ac o in isible o
he naked eye. Howe e , Mic oSpli was s ill able o unmix hese channels a a easonable quali y. One can
obse e he supe imposed inpu , he wo a ge s, and he p edic ions o Mic oSpli in Figu e S8.
Fo a second example, we inspec Task II which uses he HT-P23A da ase . He e, he wo s uc u es
a e mic o ubules and nuclei, wi h he la e being he weake channel. Simila ly o he i s example, he
weake nucleus channel is e ec i ely in isible in supe imposed images, as can be seen in Figu e S9. Howe e ,
unlike in he p e ious example, he s uc u e o nuclei in his da ase is less egula and mo e a iable.
Due o his and in ligh o he high unce ain y caused by he highly skewed channel in ensi ies, Mic oSpli
MMSE p edic ions o he nucleus channel become a he blu y, as can be seen in Figu e S9. S ill, in many
p ac ical applica ions, i.e. e e y ime he de ailed ex u e o he nuclei is no impo an , such p edic ions
can s ill be su icien o he desi ed da a analysis o be ca ied ou .
B.5 Su icien La e al Image Con ex
Ou me hod combines he bene i s o µSpli and denoiSpli . One o he bene i s o µSpli a chi ec u e we
demons a ed in [6] is i s abili y o u ilize addi ional su ounding spa ial con ex o a gi en supe imposed
inpu pa ch h ough LC inpu s and i s abili y o employ much deepe a chi ec u es. In [6], we showed ha
ha ing su icien spa ial con ex helps wi h ela i ely la ge s uc u es spanning hund eds o pixels.
C Addi ional Expe imen s
C.1 Mic oSpli s. PICASSO
In his sec ion, we compa e he pe o mance o Mic oSpli wi h PICASSO [20] (see also Figu e S4). Fo
seman ic unmixing kchannels Mic oSpli needs as inpu a single supe imposed image om he mic oscope
whe eas PICASSO needs kimages om he mic oscope which co espond o kspec ally o e lapping luo-
opho es. Due o his misma ch in he da a equi emen , a di ec compa ison is no easible. So, o compa e
Mic oSpli wi h PICASSO, we gene a e syn he ic inpu s using ou HT-LIF24 da ase . Howe e , we a gue
ha ou way o gene a ion does no deg ade he pe o mance o PICASSO bu ins ead, i should be easie
o PICASSO o p edic on his da a as compa ed o he eal da a.
Since luo opho es can be o de ed acco ding o he wa eleng h o he maximum in ensi y in hei emission
spec a, we i s de ine such an o de o ou s uc u e ypes. Nex , o gene a ing e e y channel o he inpu
o PICASSO, we de ine h ee weigh s and ake he weigh ed a e age o he h ee s uc u es using hese
weigh s. The weigh s a e se acco ding o he o de o he s uc u es se abo e. Fo example, o he i s
channel, he weigh gi en o he second s uc u e will be highe han he weigh gi en o he hi d s uc u e.
We gene a e wo se s o weigh s, one being ha de han he o he . In he ha d case, he dominan s uc u e
ype is gi en 1.5 imes mo e weigh han he nex dominan ype and 3 imes mo e weigh han he leas
dominan ype. In he easy e sion, he dominan s uc u e ype is gi en 2.5 imes mo e weigh han he
nex dominan ype and 5 imes mo e han he leas dominan ype. Once he inpu channels a e gene a ed,
we add Gaussian noise o σ= 500 o each channel independen ly. We also ain Mic oSpli on his da a
wi h he same Gaussian noise applied on op o he da a. In Figu e S4, we show he esul s on one andom
ame. In Table ST6 and Table ST7, we show he quan i a i e esul s.
C.2 T aining Mode I s T aining Mode II - How Impo an a e Spa ial
Co ela ions?
In his expe imen , we inspec he pe o mance d op be ween da a acquisi ion ypes I and II. In cells, he
loca ion o di e en s uc u es is o en qui e co- ela ed wi h one ano he . Fo example, ac in is mos ly con-
cen a ed on he cell pe iphe y whe eas he nucleus is ypically ound a he cen e o he cell. In acquisi ion
modes I and II, inpu is c ea ed by summing he c ops om indi idual channels. In acquisi ion ype I, since
he channels a e independen ly acqui ed, summing he c ops o hese di e en channels will gene a e an
inpu pa ch whe e he na u ally occu ing co-loca ion p ope y canno be p ese ed. In acquisi ion ype II,
since bo h channels a e concu en ly acqui ed, he inpu s a e c ea ed om summing he c ops o di e en
23
Task Da ase Syn he ic Noise PSNR Mic oMS-SSIM
XXIII HHMI-D258bi - 22.5 31.3 24.3 0.840 0.768 0.793
XXXI HHMI-D258bi ,denoised - 37.9 35.2 38.6 0.990 0.903 0.974
XXXII HHMI-D258bi ,denoised σ= 20, λ = 30 31.8 27.3 32.4 0.939 0.664 0.859
XXXIII HHMI-D2516bi - 23.2 27.9 24.8 0.772 0.849 0.779
XXXIV HHMI-D2516bi σ= 2K, λ = 5K23.2 27.9 24.8 0.827 0.861 0.778
XXXV HHMI-D2516bi σ= 4K, λ = 10K23.0 27.4 24.6 0.746 0.838 0.700
XXXVI HHMI-D2516bi ,0.25 - 32.4 32.2 35.0 0.990 0.929 0.991
Table ST1 Pe o mance o Mic oSpli on sub-da ase s o HHMI-25. This able p esen s
quan i a i e esul s o Mic oSpli ac oss a ious asks de ined on pa s o he HHMI-25 da ase . While
he o iginal Task XXIII on HHMI-D258bi does no lead o sa is ying p edic ions, mainly o channel 3
(see Supplemen a y Figu e S57), asks on simila da a wi h highe SNR (Task XXXI, see
Supplemen a y Figu e S61, ow 3), inc eased pixel di e si y (Task XXXIII, see Supplemen a y
Figu e S62, ow 2), o bo h (Task XXXVI, see Supplemen a y Figu e S59) demons a e no ably
imp o ed seman ic unmixing pe o mance. Tasks XXXII, XXXIV, and XXXV a e iden ical o Tasks
XXXI and XXXIII, espec i ely, bu wi h added syn he ic noise, simply o demons a e how he lowe
SNR d ops he unmixing pe o mance achie able wi h Mic oSpli .
Channel 1 Channel 2
Skew Lase Powe Lase Powe
High Mid Low High Mid Low
Balanced 24.3 23.1 21.9 29.9 24.3 23.0
Mid 28.2 24.0 22.9 25.6 22.3 21.7
High 25.2 24.3 23.3 24.1 22.4 22.8
Table ST2 Va ying he lase powe and skew wi h
Pa ia-P24 da ase . Skew column deno es he ela i e
impo ance gi en o channel 1. High skew means la ge
lase powe alloca ed o Channel 1 compa ed o Channel 2
channels wi h each c op aken om he same loca ion in he mic og aphs and he e o e hese inpu s p e-
se e he na u ally occu ing co-loca ion p ope y. In his expe imen we quan i y he bene i o using his
co-loca ion in o ma ion.
We wo k wi h he HT-T24 da ase which alls unde T aining Mode III . We ain h ee models. In
he i s model, we c ea e he inpu using T aining Mode I , ha is, we c ea e he inpu by picking a ge
pa ches om he same loca ion and he e o e main ain he spa ial co- ela ion. In he second model, we use
T aining Mode II , meaning ha we pick c ops om andom loca ions om he di e en channels and use
hem o c ea e he inpu . This model na u ally does no ha e access o na u ally occu ing spa ial co- ela ion
in o ma ion in i s aining da a. The hi d model is ained using T aining Mode III . We e alua e all h ee
models on he held-ou es se whe e he inpu s ha e spa ial co- ela ion p ese ed and a e no syn he ic, ha
is, hey a e imaged om he mic oscope. In Table ST3, we ind ha he i s model ou pe o ms he second
by 1.3 CARE-PSNR and 0.009 Mic oMS-SSIM. Na u ally, he model ained wi h T aining Mode III is
bes and ou pe o ms T aining Mode I by 0.6 CARE-PSNR and 0.004 Mic oMS-SSIM.
C.3 T aining Mode I s.T aining Mode III- Summed s. Acqui ed Inpu s
He e, we quan i y how much he pe o mance deg ades i , du ing aining, he inpu is c ea ed by simply
summing he wo channels as compa ed o inpu coming di ec ly om he mic oscope. We ind ha while
he e is indeed a pe o mance d op as can be seen in Table ST3, he d op is no de imen al. This expe imen
shows he u ili y o ou app oach in he case when syn he ic inpu is used o i s aining bu o e alua ion,
inpu s coming di ec ly om he mic oscope a e used.
C.4 Mic oSpli enables a mo e e ec i e use o he a ailable pho on budge
By il e ing ewe pho ons: T adi ional mul iplexed imaging elies on emission il e s ha selec i ely pass
pho ons om one luo opho e a a ime in o de o minimize spec al o e lap. As a consequence, a subs an ial
ac ion o emi ed pho ons is disca ded, and elaxing he il e s leads o bleed h ough a i ac s. Mic oSpli
changes his ade-o because mul iple s uc u es can be imaged in he same acquisi ion. This allows
mic oscopis s o use subs an ially b oade emission il e s, collec ing pho ons om se e al luo opho es
simul aneously, wi hou in oducing he ambigui ies ha would o he wise a ise in a mul i-channel se ing.
To oughly quan i y his ad an age, we analyzed h ee luo opho es om he HT-LIF24 da ase (DAPI,
FITC, TRITC), using hei emission spec a (downloaded om pbase.o g) no malized as p obabili y mass
24
Model PSNR SSIM
T aining Mode I: inpu = C1+C235.9 0.956
T aining Mode II: inpu = C1+C2(shu led) 34.6 0.947
T aining Mode III: inpu comes om mic oscope 36.5 0.960
Table ST3 Pe o mance compa ison o di e en Acquisi ion Modes.
We use he Sox2 s Golgi spli ing ask o he HT-T24 da ase o
his pu pose. Fo Acquisi ion Mode I and II, eal inpu image is no
used du ing aining. Ins ead, inpu is c ea ed by syn he ically
summing he wo a ge channels. In all cases, e alua ion is done on
he held-ou es se using he eal inpu channel, ha is, on he
inpu which is no syn he ic and comes di ec ly om he mic oscope.
C1 C2
(2D) Z=1 Z=5 Z=9 Z=15 (2D) Z=1 Z=5 Z=9 Z=15
CARE-PSNR 35.8 38.7 39.5 39.7 30.9 33.7 34.5 34.7
Mic oSSIM 0.865 0.878 0.885 0.886 0.729 0.757 0.772 0.767
Mic oS3IM 0.950 0.970 0.973 0.974 0.929 0.950 0.956 0.956
Table ST4 Pe o mance imp o emen wi h 3D models on HT-H24 da ase . As we inc ease
he numbe o z-slices ed o he model, we see he pe o mance imp o e in all ou me ices.
Se I Se II
C1C2C3C1C2C3
Inpu C10.50 0.33 0.17 0.625 0.25 0.125
Inpu C20.33 0.50 0.33 0.25 0.625 0.25
Inpu C30.17 0.33 0.50 0.125 0.25 0.625
Table ST5 We use he weigh s o mix he h ee channels
C1,C2and C3.
unc ions. We compa ed he ac ion o pho ons ansmi ed by con en ional mul i-colo il e con igu a ions
o he pho ons cap u ed when using a b oad, highly pe missi e il e sui able o Mic oSpli . Ac oss h ee
ep esen a i e scena ios, wi h emission il e h esholds chosen o (i) maximize pho on collec ion, (ii) educe
bleed h ough by 25%, and (iii) educe bleed h ough by 50%, wi h con en ional mul iplexed imaging being
espec i ely 22%, 34%, and 55% less pho ons e icien han he esul s ob ained using Mic oSpli (see op
hal o Supplemen a y Figu e S6).
In p ac ical e ms, his means ha e en in bleed h ough-op imized mul iplexed imaging, each channel
disca ds a la ge ac ion o emi ed pho ons, whe eas Mic oSpli can eclaim much o his loss by agg ega ing
pho ons om mul iple luo opho es in a single measu emen .
By enabling gen le imaging due o denoising: Ou me hod, nex o pe o ming seman ic unmix-
ing, also pe o ms unsupe ised denoising. Since denoising imp o es he signal- o-noise (SNR) o images
i is applied o, mic oscopis s can acqui e he aw da a mo e gen le, accep ing a lowe ini ial SNR [3]. To
illus a e his on a conc e e example, we assessed he simila i y o a biological s uc u e imaged a a ious
exposu e imes be ween 2ms and 20ms wi h e y high SNR da a o he same egions o in e es acqui ed
a 500ms exposu e ime. Mo e conc e ely, we ha e conduc ed hese expe imen s on he 3 channel da a
(Nucleus, Mic o ubules, Kine oco e) o he HT-LIF24 da ase . As shown in Supplemen a y Figu e S6 (bo -
om), e en he denoised 2ms exposu e mic og aphs lead o a conside ably highe quali y w. . . he 500ms
images han e en he 5ms aw da a, o Channel 1 e en compa ed o he 20ms aw acquisi ions, sugges ing
a leas a 3 o 10- old educ ion o he equi ed pho on budge and he e o e enabling use s o Mic oSpli
o image conside ably mo e gen le o each he same quali y equi ed o downs eam-p ocessing.
D De ails on Unce ain y Quan i ica ion and Calib a ion
In his sec ion, we de ail ou unce ain y es ima ion and calib a ion module. Ou app oach builds on he
me hodology o denoiSpli [7], o iginally inspi ed by [15], wi h a modi ica ion o he calib a ion p ocess
as desc ibed below. Ou a ia ional app oach is inhe en ly capable o sampling, meaning i can gene a e
sligh ly di e en ou pu s o he same inpu each ime a p edic ion is made. We le e age his p ope y
o p oduce mul iple p edic ions o a single inpu , esul ing in se e al p edic ed alues pe pixel. Using
hese, we calcula e he s anda d de ia ion o each pixel. To ensu e hese pixel-wise s anda d de ia ions
co espond closely o he ac ual p edic ion e o s, we apply a s aigh o wa d linea scaling. Op ing o
25
Sample p epa a ion
Human BJ ib oblas cells we e cul u ed in high-glucose DMEM (Li e Technologies, 10569) supplemen ed
wi h 10% e al bo ine se um (Li e Technologies, 26140) and penicillin-s ep omycin. Cells we e main ained in
a humidi ied incuba o a 37 deg ees Celsius wi h 5% ca bon dioxide. P io o imaging, li e cells we e washed
wi h PBS (Li e Technologies, 15140) and ypsinized using 2.5 mL o 0.05% ypsin (Li e Technologies,
25300) a 70–80% con luence. The cells we e hen ans e ed o 35 mm glass-bo om dishes (Ma Tek,
P35G-1.5-14-C) o mic oscopy.
The s aining solu ion was p epa ed by dilu ing Tubulin T acke Deep Red (The mo Fishe Scien i ic,
T34077) o 1x, CellMask O ange Ac in T acking S ain (The mo Fishe Scien i ic, A57247) o 2x, Mi o-
T acke G een FM (The mo Fishe Scien i ic, M7514) o 100 nM, and Hoechs 34580 (The mo Fishe
Scien i ic, H21486) o 5µg/mL in g ow h media. Fo imaging, cells we e incuba ed wi h 1mL o he s ain-
ing solu ion o 30 minu es a 37 deg ee Celsius, insed i e imes wi h Fluo oB i e DMEM (The mo Fishe
Scien i ic, A1896701), and subsequen ly imaged and analyzed in Fluo oB i e DMEM.
Mul i-colo s uc u ed illumina ion mic oscopy (SIM) supe - esolu ion imaging
A cus om-buil s uc u ed illumina ion mic oscope (SIM) was used o mul i-colo imaging1. Exci a ion
wa eleng hs o 642nm (Spec a-Physics, Excelsio One 642), 532nm (Spec a-Physics, Millennia V), 488nm
(Spec a-Physics, Excelsio One 488), and 405nm (Spec a-Physics, Excelsio One 405) we e employed o
exci e Tubulin T acke , CellMask O ange, Mi oT acke , and Hoechs , espec i ely. The ou lase beams
we e combined using h ee dich oic mi o s (Sem ock, Di03-R405- 1; Sem ock, Di03-R488- 1; Tho labs,
DMSP550R) and subsequen ly expanded by 4x. The combined beams we e i s di ac ed in o monoch o-
ma ic beams by a blazed g a ing (Tho labs, GR13-0605) and hen ecombined a he plane o a digi al
mic omi o de ice (DMD; Texas Ins umen s, DLP9000X VIS WQXGA). The DMD-gene a ed pa e ns
we e p ojec ed on o he sample plane h ough an objec i e lens (Nikon, SR Plan Apo, 60x, 1.27 WI).
A mul i-band dich oic mi o (Sem ock, Di01-R405/488/532/635-25x36) and an emission il e (Sem ock,
FF01-446/510/581/703-25) we e used o sepa a e exci a ion and emission ligh . Addi ionally, a 2×beam
expande was employed o u he magni y he emission signal, esul ing in a o al magni ica ion o 120×.
Fluo escence images we e cap u ed using an sCMOS de ec o (Pho ome ics, Kine ix).
Fo SIM supe - esolu ion imaging, s iped bina y pa e ns wi h a second-o de spa ial equency o
2.86µm−1a he sample plane we e displayed on he DMD. Pa e ns a h ee di e en angles wi h six phase
shi s each we e sequen ially p ojec ed, wi h an exposu e ime o 100 ms pe pa e n. Supe - esolu ion SIM
econs uc ion was pe o med using Fai SIM 2, an ImageJ-based open-sou ce so wa e. Each aw image
s ack o 2048 ×2048 ×18 was econs uc ed in o 4096 ×4096 supe - esolu ion images, whe e each pixel
co esponds o 27 nm.
F.7 The HHMI-D25 da ase
Animal Expe imen s:
He e ozygous PhAMexcised emale mice ca ying he Mi o-Dend a2 ansgene we e gene a ed h ough in-
house b eeding: i s , PhAMexcised he e ozygous males and emales (s ain #018397, Jackson Labo a o ies)
we e c ossed o de i e homozygous males, which we e hen b ed wi h wild- ype C57BL/6J emales. Mice
we e housed in sound-a enua ed, empe a u e- and humidi y-con olled ooms unde a 12-hou ligh /da k
cycle, wi h ood and wa e p o ided ad libi um. All p ocedu es we e conduc ed in acco dance wi h NIH
guidelines and we e app o ed by he Ins i u ional Animal Ca e and Use Commi ee (IACUC) a he Janelia
Resea ch Campus (P o ocol #22-0229.04). Li e s we e collec ed ia ca diac pe usion: i s wi h 1×PBS o
emo e blood, ollowed by 30 mL o 4% pa a o maldehyde (PFA) a a low a e o 2.5 mL/min o minimize
endo helial damage. Tissues we e pos - ixed in 4% PFA o 24 hou s, insed h ee imes in 1×PBS, and
s o ed in 1% PFA un il u he p ocessing.
Immunos aining and Image Acquisi ion:
Fo imaging, samples we e embedded in 4.6% low-mel ing-poin aga ose and sec ioned in o 120 µm slices
using a Leica VT 1200S ib a ome. Sec ions we e blocked in 10% e al bo ine se um wi h 0.5% T i on X-100
o 1 hou and incuba ed wi h p ima y an ibodies a 4◦C o 48 hou s. Mouse an i-PMP70 (Millipo eSigma,
SAB4200181, 1:75) and abbi an i-LAMP1 (Abcam, AB208943, 1:50) we e used o label pe oxisomes and
lysosomes, espec i ely. Seconda y an ibodies — Alexa Fluo 647 goa an i-mouse (The mo Fishe , A21235,
1:500) and Alexa Fluo 750 goa an i- abbi (The mo Fishe , A21039, 1:500) — we e applied o e nigh a
32

4◦C. Addi ional ma ke s included Alexa Fluo Plus 555 Phalloidin (The mo Fishe , A30106, 1:100) and
HCS LipidTox Red (The mo Fishe , H34476, 1:100) o label ac in and lipid d ople s, espec i ely. Nuclei
we e s ained wi h DAPI (The mo Fishe , D3571, 1:1000 dilu ion o 1 mg/mL s ock) du ing a 25-minu e
PBS wash he ollowing day. Sec ions we e clea ed using EasyIndex (Li eCan as Technologies, EI-500-1.52)
by incuba ing samples i s in 50% EasyIndex o 1 hou , ollowed by 100% EasyIndex o 3–5 hou s, and
moun ed using Secu e-Seal space s (The mo Fishe , 0523073). Imaging was pe o med on a Leica S ella is
8 con ocal mic oscope using a 63×/1.40 NA oil imme sion objec i e. Images we e acqui ed a 2048 ×2048
pixels using bidi ec ional scanning, 2×op ical zoom, and a pinhole size o 0.5 Ai y uni s (AU). A o al
dep h o 10µm was cap u ed ac oss 50 z-sec ions. Fluo opho es we e imaged using wo acquisi ion lines:
he i s included mi ochond ia, ac in, and lysosomes, while he second included nuclei, lipid d ople s, and
pe oxisomes.
G Fu he De ails on all Expe imen s (Lea ning Tasks)
In his sec ion, we men ion speci ic de ails abou indi idual asks men ioned in he Table 1. Unless explic-
i ly speci ied, we use he same hype pa ame e s o all asks. Please e e o he code o de ails on he
hype pa ame e s.
G.1 Two-channel seman ic unmixing asks
Task I
I is c ea ed om he HT-H24 da ase . I wo ks on 3D z-s acks. The acquisi ion mode used in his da ase
was T aining Mode I . We show one quali a i e example in Supplemen a y Figu e S18.
Task II
I is c ea ed om HT-P23A da ase . I wo ks on 3D z-s acks. The acquisi ion mode used in his da ase
was T aining Mode I . We show one quali a i e example in Supplemen a y Figu e S19.
Task III
I is c ea ed om he HT-P23B da ase . I wo ks on 3D z-s acks. The acquisi ion mode used in his da ase
was T aining Mode I . We show one quali a i e example in Supplemen a y Figu e S20.
Task IV-XII
These asks a e c ea ed om he Pa ia-P24 da ase . The acquisi ion mode used in hem was
T aining Mode III-a. These asks wo k on 2D ames. The e a e nine di e en acquisi ions wi hin his
da ase ha di e in (a) he lase powe dis ibu ion among he wo a ge channels and (b) he o e all
SNR. SNR has h ee le els, namely low, mid, and high, wi h low ha ing he lowes SNR and high ha ing
he highes SNR le el. The SNR le el ixes he o al combined lase powe used o bo h channels. Wi hin
a single SNR le el, we dis ibu e he powe among he wo channels in h ee ways, namely, 50/50,66/33,
and 84/16 deno ing he pe cen age lase powe alloca ion o he espec i e channel. So, 84/16 means 84%
o he o al lase powe was alloca ed o channel 1 and 16% was alloca ed o channel 2. We show one
quali a i e example pe ask in Supplemen a y Figu es S27,S29,S28,S30,S31,S32,S33,S34,S35.
Task XIII
This ask was c ea ed om he HT-T24 da ase . I wo ks on 2D ames. The acquisi ion mode used in hem
was T aining Mode III-a. We show one quali a i e example in Supplemen a y Figu e S36.
Task XIV
This ask was c ea ed om he HT-LIF da ase . I used (Nucleus) and Mic o ubules as he wo channels. I
wo ks on 2D ames. The acquisi ion mode used in hem was T aining Mode III-a. This ask wo ked wi h
he sub-da ase ha had he exposu e du a ion o 5ms. We show one quali a i e example in Supplemen a y
Figu e S37.
33
Task XV-XX
These asks we e c ea ed om he Chicago-Sch23 da ase . The Chicago-Sch23 da ase has ou s uc u es,
and hese asks cap u e all possible 2-channel seman ic unmixing asks om hese ou s uc u es. These
a e S uc u ed Illumina ion Mic oscopy (SIM) images, and due o he compu a ional pos -p ocessing ha
happens in SIM, he esul ing images ha e e y di e en noise cha ac e is ics. The e o e, we disabled he use
o noise models o all asks gene a ed om he Chicago-Sch23 da ase . Mo e speci ically, in Equa ion 2, we
se w= 0. We show one quali a i e example pe ask in Supplemen a y Figu es S21,S22,S23,S24,S25,S26.
G.2 Th ee-channel seman ic unmixing asks
Task XXI
This ask is a 3-channel seman ic unmixing ask gene a ed om he CBG-Z18 da ase . This is a 3D da ase
and so, we employ he 3D e sion o Mic oSpli . We show one quali a i e example in Supplemen a y
Figu e S16.
Task XXII
This ask is a 3-channel seman ic unmixing ask gene a ed om he CBG-N18 da ase . This is a 3D da ase
and so, we employ he 3D e sion o Mic oSpli . We show one quali a i e example in Supplemen a y
Figu e S17.
Task XXIII
This ask is a 3-channel 3D seman ic unmixing ask gene a ed om HHMI-D25 da ase . Speci ically, mi o-
chond ia, lysosomes and nuclei channels om HHMI-D258bi sub-da ase is used o c ea e his ask. This is a
3D da ase and so, we employ 3D e sion o Mic oSpli . We show one quali a i e example in Supplemen a y
Figu e S57.
Tasks XXIV-XXVIII
These asks a e gene a ed om he HT-LIF24 da ase which has ou s uc u es. The h ee s uc u es
picked o hese asks a e Nucleus, Mic o ubules and Kine oco e. While hese asks aim a spli ing apa
he abo e-men ioned h ee s uc u es, hey di e in he exposu e du a ion o he aining and e alua ion
da ase . The exposu e du a ion used is men ioned in Task De ails column o Table 1.
We show one quali a i e example pe ask in Supplemen a y Figu es S13,S14,S15,S11,S12.
Task XXXI, XXXII
These wo asks a e 3-channel 3D seman ic unmixing asks gene a ed om he denoised e sion o HHMI-
D258bi da ase , wi h denoising done by Noise2Void [4]. Mi ochond ia, lysosomes and nuclei channels om
HHMI-D258bi sub-da ase a e used. This is a 3D da ase and so, we employ 3D e sion o Mic oSpli .
While he ask XXXI wo ks di ec ly on he abo e-men ioned da a whe eas ask XXXII adds Gaussian and
Poisson noise on op o he indi idual channels. Please e e o Supplemen a y sub-sec ion B.1.1 on how he
noise was added. On a echnical node, since he ask XXXI di ec ly wo ks on he denoised da a, he e is
no a ionale o use noise models on such da a. Hence he denoiSpli loss componen is comple ely disabled
(w= 0 in Equa ion 1) and he Task XXXI is e ec i ely ained wi h µSpli con igu a ion. We show one
quali a i e example o each ask in Supplemen a y Figu e S61.
Task XXXIII-XXXV
These h ee asks a e 3-channel 3D seman ic unmixing asks gene a ed om HHMI-D2516bi da ase . Mi o-
chond ia, lysosomes and nuclei channels om HHMI-D2516bi sub-da ase a e used. This is a 3D da ase
and so, we employ 3D e sion o Mic oSpli . While he ask XXXIII wo ks di ec ly on he abo e-men ioned
da a whe eas asks XXXIV and XXXV adds Gaussian and Poisson noise on op o he indi idual chan-
nels. Please e e o Supplemen a y sub-sec ion B.1.1 on how he noise was added. We show one quali a i e
example o each ask in Supplemen a y Figu e S62.
Task XXXVI
This ask is a 3-channel 3D seman ic unmixing ask gene a ed om HHMI-D2516bi ,0.25 da ase . Mi ochon-
d ia, lysosomes and nuclei channels om HHMI-D2516bi ,0.25 sub-da ase a e used. This is a 3D da ase and
34
T aining Mode Co ela ion
be ween
s uc u es
p ese ed?
Inpu da a
is. . .
No e
T aining Mode I Yes . . . mixed
compu a-
ionally.
Exis ing imaging da a can be used (no special
acquisi ions equi ed).
T aining Mode II No . . . mixed
compu a-
ionally.
Exis ing imaging da a can be used (no spe-
cial acquisi ions equi ed). Resul s migh be o
lesse quali y i eliable s uc u al co ela ions
be ween channels o be unmixed do exis .
T aining Mode III Yes . . . imaged
di ec ly.
Mixed inpu mus di ec ly be imaged along
wi h he indi idual a ge channels. Can lead
o bes pe o mance when imaging is done
well, bu equi es addi ional mic oscopy wo k.
Table ST11 O e iew o T aining Modes: P ope ies and key ad an ages/ disad an ages o he aining modes we
p opose. We say ha he co ela ion be ween s uc u es is p ese ed when hei ela i e posi ioning in he compu a ionally
mixed inpu main ains hei biologically occu ing ela i e posi ioning in he sample.
so, we employ 3D e sion o Mic oSpli . We show one quali a i e example o he ask in Supplemen a y
Figu e S59.
G.3 Fou -channel seman ic unmixing asks
Task XXIX
This ask is gene a ed om he HT-LIF24 da ase and uses all ou s uc u es o c ea e his ask. The
exposu e du a ion used o his ask is 5ms. We show one quali a i e example pe ask in Figu e 1 .
Task XXX
This ask is gene a ed om he Chicago-Sch23 da ase and uses all ou s uc u es o c ea e his ask. Due
o he same eason as desc ibed o Tasks XV-XX, we ha e disabled he Noise model o his ask as well.
We show one quali a i e example in Supplemen a y Figu e S10.
35
Task
Idx Da ase Task De ails 2D/3D PSNR Mic oMS-SSIM
C1 C2 C1 C2
I HT-H24 - 3D 0.09 0.55 0.973 0.956
II HT-P23A - 3D 0.53 0.45 0.016 0.006
III HT-P23B - 3D 0.29 0.39 0.005 0.019
IV Pa ia-P24 high, 50/50 2D 0.17 1.1 0.010 0.009
V Pa ia-P24 high, 66/33 2D 1.83 0.19 0.045 0.002
VI Pa ia-P24 high, 84/16 2D 0.35 1.44 0.016 0.049
VII Pa ia-P24 mid, 50/50 2D 0.26 0.55 0.005 0.013
VIII Pa ia-P24 mid, 66/33 2D 0.25 0.09 0.009 0.003
IX Pa ia-P24 mid,84/16 2D 0.02 0.42 0.005 0.006
X Pa ia-P24 low, 50/50 2D 0.41 0.42 0.010 0.012
XI Pa ia-P24 low, 66/33 2D 0.02 0.28 0.006 0.006
XII Pa ia-P24 low, 84/16 2D 0.72 0.76 0.025 0.018
XIII HT-T24 - 2D 0.66 0.52 0.005 0.004
XIV HT-LIF24 - 2D 0.66 1.21 0.003 0.004
XV Chicago-Sch23 C0 s C1 2D 1.80 0.92 0.003 0.001
XVI Chicago-Sch23 C0 s C2 2D 1.68 1.32 0.005 0.002
XVII Chicago-Sch23 C0 s C3 2D 1.20 0.54 0.000 0.000
XVIII Chicago-Sch23 C1 s C2 2D 0.89 1.46 0.001 0.001
XIX Chicago-Sch23 C1 s C3 2D 0.99 0.81 0.000 0.001
XX Chicago-Sch23 C2 s C3 2D 1.01 0.82 0.000 0.000
Task
Idx Da ase Task
De ails 2D/3D PSNR Mic oMS-SSIM
C1 C2 C3 C1 C2 C3
XXI CBG-Z18 - 3D 0.12 0.18 0.12 0.002 0.001 0.001
XXII CBG-N18 - 3D 0.18 0.21 0.06 0.000 0.000 0.000
XXIII HHMI-D258bi - 3D 0.018 0.035 0.027 0.000 0.002 0.001
XXIV HT-LIF24 2ms 2D 1.19 0.90 0.66 0.007 0.003 0.004
XXV HT-LIF24 3ms 2D 1.30 0.96 0.75 0.008 0.003 0.004
XXVI HT-LIF24 5ms 2D 1.11 0.89 0.53 0.004 0.002 0.004
XXVII HT-LIF24 20ms 2D 0.85 0.62 0.73 0.002 0.001 0.002
XXVIII HT-LIF24 500ms 2D 0.61 0.35 0.66 0.001 0.001 0.001
XXXI HHMI-D258bi ,denoised - 3D 0.073 0.11 0.151 0.000 0.001 0.001
XXXII HHMI-D258bi ,denoised σ= 20, λ = 30 3D 0.076 0.094 0.153 0.000 0.003 0.002
XXXIII HHMI-D2516bi - 3D 0.03 0.126 0.016 0.002 0.003 0.001
XXXIV HHMI-D2516bi σ= 2K, λ = 5K3D 0.031 0.13 0.014 0.002 0.003 0.001
XXXV HHMI-D2516bi σ= 4K, λ = 10K3D 0.033 0.133 0.016 0.002 0.004 0.001
XXXVI HHMI-D2516bi ,0.25 - 3D 0.082 0.103 0.021 0.000 0.001 0.000
Task
Idx Da ase 2D/3D PSNR Mic oMS-SSIM
C1 C2 C3 C4 C1 C2 C3 C4
XXIX HT-LIF24 2D 0.78 1.05 0.86 0.53 0.004 0.004 0.002 0.006
XXX Chicago-Sch23 2D 1.57 0.87 1.24 0.65 0.006 0.002 0.003 0.008
Table ST12 S anda d e o alues o all en ies o Table 1.
36
Fig. S2 Seman ic unmixing o simila s uc u es wi h Mic oSpli . We show how he same s uc u e in wo a -
ge channels can s ill be unmixed, e en i he only s uc u al di e ence is a con ollable scaling ac o . See Supplemen a y
Sec ion B.3 o a de ailed desc ip ion o he conduc ed expe imen s.
37

Fig. S3 Seman ic unmixing o simila s uc u es wi h Mic oSpli . He e we plo Valida ion PSNR s. aining s eps
o all expe imen s shown in Figu e S2. Obse e ha , as he magni ica ion ac o ge s close o 1, i akes longe o he ne wo k
o ini ia e he lea ning, and he quali y o he spli ing con e ges o an o e all lowe le el. See Supplemen a y Sec ion B.3 o
a de ailed desc ip ion o he conduc ed expe imen s.
38
Fig. S4 Compa ison wi h PICASSO on a 3-channel unmixing ask. Please no e ha Mic oSpli dis inguishes i sel om
PICASSO [20] by only equi ing a single supe imposed image as inpu , a he han mul iple images wi h di e en spec al
o e laps. He e, we used he high-SNR da a om he HT-LIF24 da ase , p epa ed inpu s ha a e sui able o PICASSO, and
added addi ional (syn he ic) noise o demons a e how di e en bo h sys ems deal wi h noisy da a. In ow 1, we show example
images om he he h ee inpu channels used o PICASSO. In ows 2 and 3, we show p edic ion by PICASSO and Mic oSpli ,
espec i ely. In he las ow, we show he high-SNR g ound u h o isual e alua ion. We obse e ha , in he p esence o
noise, Picasso s a s o be challenged o unmix he da a, mos ly i i s a s ha ing simila ea u es (see columns 2 and 3, whe e
PICASSO emo es da a a ound he nucleus egion seen in column 1.
39
Fig. S5 Quan i a i e e alua ions o he e ec o signal- o-noise a ios (SNR) on in-dis ibu ion and ou -o -
dis ibu ion unmixing esul s (using he HT-LIF24 da ase ). We ained Mic oSpli models on mic og aphs acqui ed
using a ange o di e en exposu e imes. No e ha he unde lying sample ROIs emain iden ical. We hen used all ained
models ( o 2ms, 3ms, 5ms, 20ms and 500ms exposu e ime da a, indica ed in he legend) and e alua ed hei seman ic
unmixing pe o mance on all exposu e imes (x-axis), espec i ely. The ows show unmixed esul s quan i ied using Mic oSSIM,
Mic oMS-SSIM, CARE-PSNR, SSIM and MS-SSIM me ics. All plo s u ilize he legend p esen ed in he plo in he i s ow,
i s column. No e ha he plo s in his igu e also demons a e ha he MS-SSIM and SSIM me ics do no wo k well on
mic oscopy da a (while Mic oSSIM and Mic oMS-SSIM show be e sensi i i y [8]).
40
400 500 600 700
Wa eleng h (nm)
0.00
0.01
0.02
Op imized o pho on collec ion (PC)
Pho ons w ongly assigned: 22%
DAPI
FITC
TRITC
400 500 600 700
Wa eleng h (nm)
0.00
0.01
0.02
Balanced (BT & PC)
Pho ons w ongly assigned + los : 34%
DAPI
FITC
TRITC
400 500 600 700
Wa eleng h (nm)
0.00
0.01
0.02
Op imized o Bleed- h ough (BT)
Pho ons w ongly assigned + los : 55%
DAPI
FITC
TRITC
GT Noisy s GT High-SNR P edic ion s GT High-SNR
Acq. PSNR Mic oMS-SSIM PSNR Mic oMS-SSIM
Du a ion C1 C2 C3 C1 C2 C3 C1 C2 C3 C1 C2 C3
2ms 23.3 25.1 26.1 0.839 0.772 0.869 31.0 32.2 36.3 0.940 0.973 0.944
3ms 23.6 26.0 27.2 0.842 0.780 0.871 30.8 32.2 36.1 0.940 0.973 0.948
5ms 24.6 28.3 29.9 0.857 0.817 0.875 32.9 34.3 37.6 0.960 0.983 0.963
20ms 30.0 35.6 38.16 0.920 0.942 0.914 37.0 39.7 41.4 0.984 0.994 0.989
Fig. S6 Pho on e icien imaging wi h Mic oSpli : a conc e e example. We quan i y wo mechanisms h ough which
Mic oSpli educes he equi ed pho on budge : (i)Reduced pho on il e ing ( op):Emission spec a o he h ee luo opho es
DAPI, FITC, and TRITC a e shown oge he wi h example emission il e bands used in con en ional mul i-colo imaging.
F om le o igh , we illus a e h ee se ings ha inc easingly ade o bleed- h ough (BT) agains pho on e iciency: a
con igu a ion ha s ongly supp esses BT, a balanced con igu a ion ha collec s mo e pho ons a he cos o some BT, and
a highly pe missi e con igu a ion ha cap u es nea ly all emi ed pho ons bu su e s no able spec al o e lap. Mic oSpli , by
con as , allows imaging mul iple s uc u es wi hin a single b oad emission band and subsequen ly eassigning he collec ed
in ensi ies o hei espec i e ou pu channels leading o he indica ed pho on-e iciency inc eases. (ii)Denoising (bo om):
Denoising enables epu posing o he a ailable pho on budge by acqui ing lowe -SNR mic og aphs, which Mic oSpli can
es o e o high-SNR p edic ions. We compa e aw da a acqui ed a 2ms, 3ms, 5ms, and 20ms exposu e imes wi h high-SNR
(500ms) e e ence images o he same egions in he HT-LIF24 da ase ( h ee channels: Nucleus, Mic o ubules, Kine ocho e).
The i s six da a columns in he able quan i y he simila i y o low-exposu e aw da a o he 500ms e e ence, while he
igh mos six columns show he co esponding simila i ies o Mic oSpli p edic ions (iden ical da a as in Table1). In all cases,
he Mic oSpli p edic ions exhibi highe quali y han he co esponding aw inpu s. S ikingly, p edic ions om 2ms exposu es
al eady su pass he quali y o 5ms aw da a o all channels, and o Channel 1 e en exceed he 20ms aw da a. In his example,
his co esponds o a leas a h ee- old educ ion in equi ed pho on budge pe acquisi ion, and likely close o an o de o
magni ude when a e aged ac oss he h ee channels.
41
Fig. S21 Quali a i e E alua ion o Task XV om Chicago-Sch23 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
48

Fig. S22 Quali a i e E alua ion o Task XVI om Chicago-Sch23 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
49
Fig. S23 Quali a i e E alua ion o Task XVII om Chicago-Sch23 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
50
Fig. S24 Quali a i e E alua ion o Task XVIII om Chicago-Sch23 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
51
Fig. S25 Quali a i e E alua ion o Task XIX om Chicago-Sch23 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
52
Fig. S26 Quali a i e E alua ion o Task XX om Chicago-Sch23 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
Fig. S27 Quali a i e E alua ion o Task IV om Pa ia-P24 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
53

Fig. S28 Quali a i e E alua ion o Task VI om Pa ia-P24 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
54
Fig. S29 Quali a i e E alua ion o Task V om Pa ia-P24 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
Fig. S30 Quali a i e E alua ion o Task VII om Pa ia-P24 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
55
Fig. S31 Quali a i e E alua ion o Task VIII om Pa ia-P24 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
Fig. S32 Quali a i e E alua ion o Task IX om Pa ia-P24 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
56
Fig. S33 Quali a i e E alua ion o Task X om Pa ia-P24 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
Fig. S34 Quali a i e E alua ion o Task XI om Pa ia-P24 da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
57
Fig. S54 Calib a ion plo o Task I om Da ase HT-H24
Fig. S55 Calib a ion plo o Task XXI om Da ase CBZ-Z18
Fig. S56 Calib a ion plo o Task XXII om Da ase CBZ-N18
64

Fig. S57 Quali a i e E alua ion o Task XXIII om HHMI-D258bi da ase . No e ha we show he a ge and he p edic ion
co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle. Also no e ha he p edic ions o
channel 3 a e o a he poo quali y and ha you can ind a desc ip ion o how his p oblem was sol ed in he Supplemen a y
Sec ion B.
Fig. S58 Quali a i e E alua ion o Task XXXIII om HHMI-D2516bi da ase . No e ha we show he a ge and he
p edic ion co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
65
Fig. S59 Quali a i e E alua ion o Task XXXVI om HHMI-D2516bi ,0.25 da ase . No e ha we show he a ge and he
p edic ion co esponding o he inpu c op which is deno ed in Inpu panel by a whi e do ed ec angle.
Fig. S60 E ec o SNR on Model Pe o mance (HT-LIF24 Da ase ): We e alua e how SNR in luences model
pe o mance using he HT-LIF24 da ase . Di e en models a e ained on da a subse s acqui ed wi h a ying exposu e du a-
ions—leading o di e en SNRs—and hei p edic ions a e compa ed o e a common egion o in e es . High- equency de ails
in he p edic ions (especially he hi d channel) a e isibly educed when he inpu SNR is lowe .
66
Fig. S61 E ec o SNR on Model Pe o mance (HHMI-D258bi Da ase ): We assess he impac o SNR on a subse
o he HHMI-D25 da ase . Compa ing he p edic ions ( ow 2, esul s o Task XXIII) wi h he g ound u h ( ow 1), we obse e
ha he p edic ion quali y, pa icula ly o he hi d channel, is no good, wi h en i e pa s o he s uc u es being pu in o he
o he channels. We hen ain Mic oSpli using a Noise2Void [4] denoised e sion o he same da a, leading o much imp o ed
seman ic unmixing pe o mance ( ow 3, Task XXXI). Finally, we e-in oduced syn he ic Gaussian and Poisson noise o he
denoised HHMI-D25 da a used in Task XXXI and e ained Mic oSpli on his lowe -SNR da a ( ow 4, Task XXXII). As i
was likely o be expec ed, his does again d op he seman ic unmixing pe o mance.
67
Fig. S62 E ec o SNR on Model Pe o mance (HHMI-D2516bi Da ase ): We assess he impac o SNR on he
HHMI-D2516bi da ase . Fo his pa o he HHMI-D25 da a, he p edic ions ( ow 2, Task XXXIII) a e isually mo e close
o he a ge images compa ed o esul s on HHMI-D258bi da ase (Figu e S61, ow 2, Task XXIII). We hen in oduce wo
le els o addi ional Gaussian and Poisson noise o he HHMI-D2516bi da a o educe SNR and e ain Mic oSpli on hose
noisie e sions o he da a ( ow 3 and 4, Tasks XXXIV and XXXV, espec i ely). As i was likely o be expec ed, he educed
SNR leads o a no iceable decline in he seman ic unmixing pe o mance.
68