METHODS
published: 10 Feb ua y 2022
doi: 10.3389/ comp.2022.777728
F on ie s in Compu e Science | www. on ie sin.o g 1Feb ua y 2022 | Volume 4 | A icle 777728
Edi ed by:
Ma cello Pelillo,
Ca’ Fosca i Uni e si y o Venice, I aly
Re iewed by:
Xinggang Wang,
Huazhong Uni e si y o Science and
Technology, China
Csaba Beleznai,
Aus ian Ins i u e o Technology (AIT),
Aus ia
*Co espondence:
Flo ian Jug
[email p o ec ed]
Special y sec ion:
This a icle was submi ed o
Compu e Vision,
a sec ion o he jou nal
F on ie s in Compu e Science
Recei ed: 15 Sep embe 2021
Accep ed: 13 Janua y 2022
Published: 10 Feb ua y 2022
Ci a ion:
A z M, Deschamps J, Schmied C,
Pie zsch T, Schmid D, Tomancak P,
Haase R and Jug F (2022) LABKIT:
Labeling and Segmen a ion Toolki o
Big Image Da a.
F on . Compu . Sci. 4:777728.
doi: 10.3389/ comp.2022.777728
LABKIT: Labeling and Segmen a ion
Toolki o Big Image Da a
Ma hias A z 1,2, Jo an Deschamps1,2,3, Ch is ophe Schmied3, Tobias Pie zsch1,2,
Debo ah Schmid 1,2,4, Pa el Tomancak1,2,5, Robe Haase 1,2,6 and Flo ian Jug 1,2,3*
1Cen e o Sys ems Biology D esden, D esden, Ge many, 2Max Planck Ins i u e o Molecula Cell Biology and Gene ics,
D esden, Ge many, 3Fondazione Human Technopole, Milan, I aly, 4Max Delb ück Cen e o Molecula Medicine, Be lin,
Ge many, 5IT4Inno a ions, VŠB-Technical Uni e si y o Os a a, Os a a, Czechia, 6DFG Clus e o Excellence “Physics o
Li e”, TU-D esden, D esden, Ge many
We p esen LABKIT, a use - iendly Fiji plugin o he segmen a ion o mic oscopy image
da a. I o e s easy o use manual and au oma ed image segmen a ion ou ines ha
can be apidly applied o single- and mul i-channel images as well as o imelapse
mo ies in 2D o 3D. LABKIT is speci ically designed o wo k e icien ly on big image
da a and enables use s o consume lap ops o con enien ly wo k wi h mul iple- e aby e
images. This e iciency is achie ed by using ImgLib2 and BigDa aViewe as well as a
memo y e icien and as implemen a ion o he andom o es based pixel classi ica ion
algo i hm as he ounda ion o ou so wa e. Op ionally we ha ness he powe o g aphics
p ocessing uni s (GPU) o gain addi ional un ime pe o mance. LABKIT is easy o ins all
on i ually all lap ops and wo ks a ions. Addi ionally, LABKIT is compa ible wi h high
pe o mance compu ing (HPC) clus e s o dis ibu ed p ocessing o big image da a.
The abili y o use pixel classi ie s ained in LABKIT ia he ImageJ mac o language
enables ou use s o in eg a e his unc ionali y as a p ocessing s ep in au oma ed image
p ocessing wo k lows. Finally, LABKIT comes wi h ich online esou ces such as u o ials
and examples ha will help use s o amilia ize hemsel es wi h a ailable ea u es and
how o bes use LABKIT in a numbe o p ac ical eal-wo ld use-cases.
Keywo ds: segmen a ion, labeling, machine lea ning, andom o es , Fiji, open-sou ce
1. INTRODUCTION
In ecen yea s, new and powe ul mic oscopy and sample p epa a ion echniques ha e eme ged,
such as ligh -shee (Huisken e al., 2004), supe - esolu ion mic oscopy (Hell and Wichmann,
1994; Gus a sson, 2000; Be zig e al., 2006; Hess e al., 2006; Rus e al., 2006), mode n issue
clea ing (Dod e al., 2007; Hama e al., 2011), o se ial sec ion scanning elec on mic oscopy (Denk
and Ho s mann, 2004; Kno e al., 2008) enabling esea che s o obse e biological issues and hei
unde lying cellula and molecula composi ion and dynamics in unp eceden ed de ails. To localize
objec s o in e es and exploi such ich da ase s quan i a i ely, scien is s need o pe o m image
segmen a ion, e.g., di iding all pixels in an image in o o eg ound pixels (pa o objec s o in e es )
and backg ound pixels.
The esul o such a pixel classi ica ion is a bina y mask, o a (mul i-)label image i mo e
han one o eg ound class is needed o disc imina e di e en objec s. Masks o label images
enable downs eam analysis ha ex ac biologically meaning ul seman ic quan i ies, such as he
numbe o objec s in he da a, mo phological p ope ies o hese objec s (shape, size, e c.), o
A z e al. LABKIT: Labeling and Segmen a ion Toolki
acks o objec mo emen s o e ime. In mos p ac ical
applica ions, image segmen a ion is no an easy ask o sol e. I
is o en ende ed di icul by he sample’s biological a iabili y,
impe ec imaging condi ions (e.g., leading o noise, blu , o o he
dis o ions), o simply by he complica ed h ee-dimensional
shape o he objec s o in e es .
Cu en esea ch in bio-image segmen a ion ocuses p ima ily
on de eloping new deep lea ning app oaches, wi h mo e classical
me hods cu en ly ecei ing li le a en ion. Algo i hms, such as
S a Dis (Schmid e al., 2018), DenoiSeg (Buchholz e al., 2020),
Pa chPe Pix (Mais e al., 2020), Plan Seg (Wolny e al., 2020),
CellPose (S inge e al., 2021), o EmbedSeg (Lali e al., 2021)
ha e con inuously aised he s a e-o - he a and ou pe o m
classical me hods in quali y and accu acy o achie ed au oma ed
segmen a ion. While hese app oaches a e e y powe ul indeed,
deep lea ning does equi e some expe knowledge, dedica ed
compu a ional esou ces no e e ybody has access o, and
ypically la ge quan i ies o densely labeled g ound- u h da a o
ain on.
Mo e classical app oaches, on he o he hand, can also yield
esul s ha enable he equi ed analysis, while o en emaining
as and easy o use on any lap op o wo ks a ion. Examples
o such me hods ange om in ensi y h esholding and seeded
wa e shed, o shallow machine lea ning app oaches on manually
chosen o designed ea u es. One c ucial p ope y o shallow
echniques, such as andom o es s (B eiman, 2001), is ha hey
equi e o de s o magni ude less g ound- u h aining da a han
deep lea ning based me hods. Hence, mul iple so wa e ools
pai hem wi h use - iendly in e aces, e.g., CellP o ile (McQuin
e al., 2018), Ilas ik (Be g e al., 2019), QuPa h (Bankhead
e al., 2017), and T ainable Weka Segmen a ion (A ganda-
Ca e as e al., 2017). The la e specializes in andom o es
classi ica ion and is a ailable wi hin Fiji (Schindelin e al., 2012),
a widely-used image analysis and p ocessing pla o m based on
ImageJ (Schneide e al., 2012) and ImageJ2 (Rueden e al.,
2017). I is, eg e ably, no capable o p ocessing e y la ge
da ase s due o i s excessi e demand o CPU memo y, lea ing
he sizable Fiji communi y wi h a lack o use - iendly pixel
classi ica ion o segmen a ion ools ha can ope a e on la ge
mul i-dimensional da a.
The equi ed ounda ions o such a so wa e ool ha e
in ecen yea s been buil by he ib an esea ch so wa e
enginee ing communi y a ound Fiji and ImageJ2. Speci ically,
he p oblem o handling la ge mul i-dimensional images
has been add essed by a gene ic and powe ul lib a y
called ImgLib2 (Pie zsch e al., 2012). Addi ionally, a
as , memo y-e icien , and ex ensible image iewe , he
BigDa aViewe (Pie zsch e al., 2015), enables ool de elope s o
c ea e in ui i e and as da a handling in e aces.
He e, we p esen an image labeling and segmen a ion
ool called LABKIT. I combines he powe o ImgLib2 and
BigDa aViewe wi h a new implemen a ion o andom o es
pixel classi ica ion. LABKIT ea u es a use - iendly in e ace
allowing o apid sc ibble labeling, aining, and in e ac i e
cu a ion o he segmen ed image. LABKIT also allows use s
o ully manually label pixels o oxels in he loaded images.
I can be easily ins alled in Fiji, and di ec ly called om i s
mac o p og amming language. LABKIT addi ionally ea u es
GPU accele a ion using CLIJ (Haase e al., 2020), and can be
used on high pe o mance compu ing (HPC) clus e s hanks o
a command-line in e ace.
2. IMAGE SEGMENTATION WITH LABKIT
LABKIT’s use in e ace is buil a ound he
BigDa aViewe (Pie zsch e al., 2015), which allows in e ac i e
explo a ion o image olumes o any size and dimension on
consume compu ing ha dwa e (Figu es 1A,B). Beyond he
common BigDa aViewe ea u es, use s ha e access o a se o
simple d awing ools o manually pain o co ec exis ing labels
on image pixels in 2D and oxels in 3D. Impo an ly, he aw
da a is ne e modi ied by any such ac ions. Pixel and oxel labels
a e g ouped by classes in indi idual laye s (e.g., backg ound,
nucleus o o ganelle). Each class is ep esen ed by a modi iable
colo , and can be used o anno a e di e en ypes o objec s and
s uc u es o in e es in he image.
Thanks o he in ui i e in e ace design, use s can e icien ly
segmen hei images by manually d awing dense labels on
he en i e image (Figu e 1C). Labels ha a e gene a ed wi h
he d awing ools can di ec ly be sa ed as images o expo ed
o Fiji o downs eam p ocessing. Dense manual labelings
o comple e images o olumes c ea ed wi h LABKIT can be
used o manually segmen objec s, as was done p e iously o
mask pa icles in c yo-elec on omog ams o Chlamydomonas
(Jo dan and Pigino, 2019).
Howe e , his p ocess is e y ime consuming and doesn’
scale well o la ge da a. LABKIT is he e o e o en used o densely
and manually label a subse o he image da a, which is hen
used as g ound- u h o supe ised deep lea ning app oaches.
Published examples include he gene a ion o g ound- u h
aining da a o a mouse and a Pla yne is da ase in o de o
segmen cell nuclei wi h EmbedSeg (Lali e al., 2021). LABKIT
is also sugges ed as a ool o choice o g ound- u h gene a ion
by o he deep lea ning me hods (Schmid e al., 2018; Buchholz
e al., 2020; Ho la a e al., 2020). S ill, manually gene a ing
su icien amoun o g ound- u h aining labels o exis ing
deep lea ning me hods emains a cumbe some and edious ask.
In o de o c ea e a high quali y segmen a ion while
main aining low use inpu , LABKIT ea u es a andom
o es (B eiman, 2001) based pixel classi ica ion algo i hm, wi h
all ea u e compu a ions op imized o quick un imes. Ins ead
o anno a ing en i e objec s, a andom o es is ained on a
ew pixel labeling pe class only. These spa se manual labels,
o sc ibbles (see Figu e 1D, le ), a e di ec ly d awn by use s
o e he image. Na u ally, sc ibbles mus be d awn on pixels
ep esen a i e o each class. Once ained, he andom o es
classi ie enables he gene a ion o a segmen a ion (dense pixel
classi ica ion, see Figu e 1D). Two o mo e classes can be
used o dis inguish o eg ound objec s om backg ound pixels.
Figu es 2A,B showcase examples o a single o eg ound and
backg ound classes. I desi ed, ou o ocus objec s can e en
be disca ded, o example by making such pixels pa o he
backg ound class (Figu e 2B, a owheads). Fo mo e complex
F on ie s in Compu e Science | www. on ie sin.o g 2Feb ua y 2022 | Volume 4 | A icle 777728
A z e al. LABKIT: Labeling and Segmen a ion Toolki
FIGURE 1 | LABKIT allows easy manual labeling and au oma ic segmen a ion o la ge image olumes: (A) Maximum in ensi y p ojec ion o a single ime poin om a
∼1 TB imelapse o a de eloping Pa hyale emb yo imaged li e wi h ligh shee mic oscopy. (B) LABKIT’s use in e ace is based on BigDa aViewe and allows isualizing
and in e ac ing wi h la ge olumes o image da a. A slice o he de eloping Pa hyale emb yo is shown. (C) Use s can label la ge da ase s wi h dense manual
anno a ions using LABKIT’s d awing in e ace. (D) A co e ea u e o LABKIT is he apid segmen a ion o la ge image da a using spa se manual labels (sc ibbles)
combined wi h andom o es pixel classi ica ion o au oma ically p oduce he inal segmen a ion. Scale ba s 100 µm(A), 50 µm(B), and 25 µm(C,D).
segmen a ion asks ha need o disc imina e a ious isible
s uc u es (e.g., nucleus s. cy oplasm s. backg ound) o cell
ypes (as in Figu e 2C), wo o mo e o eg ound classes can be
used (Figu e 2D).
As opposed o deep lea ning algo i hms, andom o es s
a e ypically ained in a ma e o seconds. D awing sc ibbles
and compu ing he segmen a ion can he e o e con enien ly be
i e a ed due o he e icien pa alleliza ion we ha e implemen ed,
leading o li e segmen a ion. Li e esul s a e compu ed and
displayed only on he cu en ly isualized image slice in
BigDa aViewe o inc ease he in e ac i i y. Hence, he e ec
o addi ional sc ibbles (spa se labels) is ins an ly isible and
use s can s op once he au oma ed ou pu o he pixel classi ie
eaches su icien quali y. This i e a i e wo k low makes wo king
wi h LABKIT e y e icien , e en when uly la ge image da a
a e being p ocessed. BigDa aViewe ’s bookma king ea u e can
addi ionally be used o quickly jump be ween p e iously de ined
image egions, he eby allowing alida ing he quali y o he
pixel classi ie on mul iple a eas. Since we use ImgLib’s caching
in as uc u e, all image blocks ha ha e once been compu ed a e
kep in memo y and swi ching be ween bookma ks o b owsing
be ween pa s o a huge olume is as and isually pleasing.
Once su icien ly ained, he classi ie can be sa ed o la e
use in in e ac i e LABKIT sessions o in Fiji/ImageJ mac os.
The en i e da ase can be di ec ly segmen ed and he esul s
sa ed o disk. Recen ly, spa se labeling combined wi h andom
o es pixel classi ica ion in LABKIT was used o segmen mice
epide mal cells (Bo nes e al., 2021), as well as mRNA oci in
neu ons (A shadi e al., 2021).
Once he image is ully segmen ed, he gene a ed
segmen a ion masks can be ans e ed o label laye s and
he d awing ools can now be used o cu a e hem. The goal o
cu a ion is o esol e he emaining e o s made by he ained
pixel classi ie , such as d awing missing pa s, illing holes,
e asing mislabeling and dele ing spu ious blobs (Figu e 3). Label
cu a ion is pe o med un il he cu a ed segmen a ion is deemed
sa is ac o y o downs eam p ocessing o analysis. LABKIT can
also be used o cu a e segmen a ion esul s ob ained by o he
me hods ha a e no a ailable wi hin LABKIT, including deep
lea ning based me hods (Jain e al., 2020).
Au oma ed segmen a ion wi h LABKIT and he possibili y
o quickly cu a e any au oma ed segmen a ion esul make
LABKIT a powe ul ool ha can conside ably sho en he ime
equi ed o gene a e g ound- u h da a o aining deep lea ning
app oaches. Fo example, we compa ed au oma ic and manual
segmen a ion wi h LABKIT on a a he small subse o images
(N=26, see one example in Figu e 4A) made publicly a ailable by
he 218 Da a Science Bowl (Caicedo e al., 2019). We segmen ed
all images wi hin 5 min by i e a i e sc ibbling and au oma ed
segmen a ion (see Figu e 4B). While many images consis ed
o homogeneous nuclei and led o high quali y esul s, images
wi h he e ogeneous nuclei esul ed in segmen a ion e o s (see
F on ie s in Compu e Science | www. on ie sin.o g 3Feb ua y 2022 | Volume 4 | A icle 777728
A z e al. LABKIT: Labeling and Segmen a ion Toolki
FIGURE 2 | Seman ic segmen a ion o mic oscopy images wi h LABKIT’s pixel classi ica ion: (A) Maximum in ensi y p ojec ion o a con ocal s ack showing HeLa cells
exp essing C1-GFP (le ), nex o he spa se labeling (sc ibbles, cen e ) and esul ing cell segmen a ion ( igh ). (B) B igh ield mic oscopy image o E.coli, spa se
labeling disc imina ing cells and backg ound and he esul ing segmen a ion. A owheads show ha segmen a ion o ou -o - ocus objec s can be educed by
including pixels o such objec s in he backg ound class. (C) Fixed mouse li e issue sec ion s ained wi h immuno luo escence and imaged in mul iple channels wi h a
spinning disk con ocal mic oscope, showing Hepa ocy e nuclei s ained wi h an ibody agains HNF-4αa ansc ip ion ac o exp essed in hepa ocy es, hepa ocy e
cy oplasm (au o luo escence) and all nuclei s ained wi h DAPI. (D) Labeling and esul ing segmen a ion o he li e issue sec ion shown in (A), segmen ing Hepa ocy e
cy oplasm (g een), Hepa ocy e nuclei (blue), nuclei o non-pa enchymal cells (yellow) and sinusoids (magen a). Scale ba s 20 µm(A),(C), and 5 µm(B).
a ows in Figu e 4B). Such e o s include spu ious ins ances
ha do no co ela e wi h any objec in he o iginal image,
ins ances ha co espond o he usion o mul iple ins ances,
ins ances wi h holes, o e en ins ances ha spli in wo. Such
e o s a e ob iously undesi able and nega i ely impac he o e all
a e age p ecision sco e (AP = 0.72, see Me hods o he me ics
F on ie s in Compu e Science | www. on ie sin.o g 4Feb ua y 2022 | Volume 4 | A icle 777728
A z e al. LABKIT: Labeling and Segmen a ion Toolki
FIGURE 3 | LABKIT labeling ools used o cu a ion: labels gene a ed by
manual dense labeling o au oma ic segmen a ion can be e icien ly cu a ed
wi h d awing, illing, e asing, o dele ion o en i e objec s. Scale ba 10 µm.
de ini ion). As desc ibed abo e, all such segmen a ion e o s
can easily be co ec ed wi hin LABKIT, ei he by adding spa se
labels co esponding o ypical a eas wi h e o s, done du ing
he i e a i e p ocess, o when hey pe sis by manually cu a ing
he esidual e o s in he inal au oma ed esul s (Figu e 4C).
Cu a ing all 26 images ook an addi ional 10 min and aised he
co esponding a e age p ecision o 0.76, a sco e e y close o he
in e -obse e dis ance (AP = 0.78), as shown in Figu es 4C,D.
In con as , manually segmen ing all images equi ed mo e han
an hou (Figu e 4D), which is ou imes longe han sc ibble-
based pixel classi ica ion wi h LABKIT, ollowed by ull cu a ion
o he esul s o ob ain images o compa able quali y.
Hence, whene e LABKIT au oma ed segmen a ion is by i sel
no su icien , manually cu a ing he esul s yields g ound- u h
da a ha can be used o ain a deep lea ning me hod, leading o
highe segmen a ion quali y wi h less labeling e o .
3. LABKIT PIXEL CLASSIFIER
LABKIT p o ides a pixel classi ica ion algo i hm o au oma ic
segmen a ion. The algo i hm uses a andom o es o classi y each
pixel independen ly in o use -de ined classes (e.g., o eg ound
and backg ound). Random Fo es s (B eiman, 2001) a e widely
used supe ised machine lea ning me hods, and as such mus
be ained on a gi en body o g ound- u h labels (p e-
classi ied example pixels). In LABKIT, he andom o es
classi ie is ained on manually labeled pixels (sc ibbles), an
app oach simila o ilas ik (Be g e al., 2019) o T ainable
Weka Segmen a ion (A ganda-Ca e as e al., 2017). As opposed
o mos implemen a ions, LABKIT’s classi ie is speci ically
op imized o be able o handle e y la ge image da a.
In a i s s ep, we compu e a ea u e ec o o each labeled
pixel. This is achie ed by applying a con igu able se o il e s
o he gi en image o images. To his end LABKIT o e s a
se o image il e s commonly used in image analysis, such
as Gaussian, di e ence o Gaussians o Laplacian il e s. Each
selec ed il e c ea es an ou pu image ha emphasizes di e en
ea u es o a gi en inpu image. Fil e esponses o each pixel
a e hen added o hei ea u e ec o . The inal ea u e ec o s
o all labeled pixels a e pai ed wi h hei espec i e g ound- u h
FIGURE 4 | Compa ing au oma ic and manual g ound- u h gene a ion wi h
LABKIT:(A) Fluo escence image o nuclei (ou o 26 images) ex ac ed om he
2018 Da a Science Bowl (Caicedo e al., 2019). (B) Resul s om LABKIT
au oma ed segmen a ion o (A) a e ex ac ing connec ed componen s and
gi ing each ins ance a unique pixel alue. The a ows poin o a ious
segmen a ion e o s. On he op igh co ne , he o al ime necessa y o ob ain
he co esponding segmen a ion o all 26 images (including labeling) is
indica ed. Below he iming is he a e age p ecision (see Me hods) as
compa ed o a dense manual labeling pe o med by ano he obse e .
(C) Cu a ion o (B) wi h same pos -p ocessing. The a ows poin o he
co ec ed e o s men ioned in (B). The iming in o ma ion includes (B).
(D) Dense manual labeling o (A) and he same pos -p ocessing as in (B). No
scale ba was a ailable o he images.
classes, oge he cons i u ing he aining se . This da a is hen
used o ain he andom o es , consis ing o 100 decision ees,
using he Fas RF lib a y (Supek, 2015).
A e aining, he andom o es classi ie can p edic pixel
classes di ec ly om he ea u e ec o o any gi en pixel. Hence,
in a inal s ep, we apply he andom o es o he ea u e ec o s o
all pixels in he en i e body o da a, he eby e ec i ely compu ing
he desi ed seman ic segmen a ion.
Since compu ing ea u e ec o s and inal andom o es
p edic ions consume by a he mos compu a ional esou ces, i
was c ucial o op imize hei un ime. To his end, we p ocess
image chunks in pa allel, wi h he chunked memo y handling
being suppo ed by ImgLib2 (Pie zsch e al., 2012). Addi ionally,
we implemen ed OpenCL ke nels, allowing us o bene i om as
GPU compu a ions (Haase e al., 2020).
4. LIMITATIONS OF THE PIXEL
CLASSIFICATION
The simplici y o he pixel classi ica ion algo i hm ensu es
e iciency, bu also exposes i o ce ain limi a ions and po en ial
F on ie s in Compu e Science | www. on ie sin.o g 5Feb ua y 2022 | Volume 4 | A icle 777728

A z e al. LABKIT: Labeling and Segmen a ion Toolki
FIGURE 5 | Pixel classi ica ion ailu e modes: (A) Image o a con ac ile acuole po e in pa amecium cauda um acqui ed wi h ansmission elec on mic oscopy (le ),
in which inne po e (blue), cy oplasm (g een) and backg ound (magen a) we e labeled wi h sc ibbles (cen e ). LABKIT’s classi ie ails o dis inguish backg ound and
inne po e classes ( igh ). Image p o ided by Richa d Allen (Cell Image Lib a y, 38894). (B) Mi o ic spindle o a P k2 cell exp essing GFP- ubulin imaged in wide- ield
luo escence (le ). Le and igh hal es o he mi o ic spindle a e labeled in di e en classes (blue and g een, espec i ely), while backg ound is labeled in magen a
(cen e ). LABKIT’s classi ie is incapable o disc imina ing simila s uc u es ( igh ). Image p o ided by Sophie Dumon and Timo hy J. Mi chison (Cell Image Lib a y,
6568). (C) B igh ield image o C. elegans (le ), labeled wi h wo classes (cen e ): o eg ound (g een) and backg ound (magen a). A e applying connec ed
componen s analysis o he classi ica ion esul , con iguous wo ms belong o he same objec ( igh ). Image p o ided by F ed Ausubel (B oad Bioimage Benchma k
Collec ion, BBBC010). Scale ba s (A) 500 nm, (B) 10 µm, and (C) 500 µm.
ailu e modes. This mainly comes om he ac ha he il e
ke nels used o compu e he ea u e ec o ha e limi ed sizes.
Wi h he de aul se ings in LABKIT, il e s espond mos ly
o a 16x16 window (2D), meaning ha decision abou he
class o a gi en pixel is based on il e esponses in a small
neighbo hood. A di ec consequence is illus a ed in Figu e 5A,
whe e LABKIT was used o segmen he image o a acuole
po e in pa amecium cauda um (le panel) using h ee classes:
inne po e (cen al panel, in blue), acuole (in g een) and
backg ound (in magen a). Because inne po e and backg ound
pixels ha e simila ex u e, he classi ie canno ell hem apa
and assigns backg ound pixels o he inne po e class, and ice
e sa.
All il e s used o calcula e he ea u e ec o s a e designed
o be ansla ion and o a ion in a ian . Hence, he algo i hm
classi ies wo objec s he same way ega dless o hei posi ion o
o ien a ion in he image. While his is a sensible assump ion o
mos applica ions on mic oscopy da a, use s should ce ainly be
awa e o his. One po en ial p oblem is showcased in Figu e 5B,
which shows a mi o ic spindle in luo escence mic oscopy (le )
and an a emp a assigning each hal o he mi o ic spindle o a
di e en class (cen al panel). Al hough bo h sides o he mi o ic
spindle a e spa ially dis inc and in di e en o ien a ions, he
classi ie ails a disc imina ing hem (Figu e 5B, igh ).
Fu he mo e, con iguous objec s canno be sepa a ed by he
andom o es classi ie . This is illus a ed in Figu e 5C, which
F on ie s in Compu e Science | www. on ie sin.o g 6Feb ua y 2022 | Volume 4 | A icle 777728
A z e al. LABKIT: Labeling and Segmen a ion Toolki
TABLE 1 | Benchma king compu a ion speed while segmen ing a la ge biological image on a ious ha dwa e: he expe imen was pe o med on a lap op wi h and
wi hou GPU accele a ion, and on di e en numbe s o CPU and GPU clus e nodes.
Ha dwa e GPU Run ime Speed-up Th oughpu in gigapixel
Lap op No 4 h 23 min 00 s 1 3.05 / h = 0.05 / min
Lap op Yes 35 min 12 s 7.5 0.38 / min
1 CPU clus e node No 1 h 08 min 10 s 1 0.20 / min
10 CPU clus e nodes No 6 min 15 s 10.9 2.14 / min
50 CPU clus e nodes No 1 min 35 s 43.1 8.45 / min
1 GPU clus e node Yes (2) 8 min 23 s 1 1.60 / min
10 GPU clus e nodes Yes (2) 1 min 03 s 7.9 12.74 / min
In each ca ego y, he speed-up is calcula ed in compa ison o he slowe en y. Numbe s in be ween pa en hesis in he GPU column indica e he numbe o GPU pe clus e node.
shows C. elegans wo ms imaged in b igh - ield mic oscopy
(le panel). While he classi ie can co ec ly dis inguish he
wo ms body om he backg ound, a connec ed componen
analysis applied o he classi ica ion esul (Figu e 5C, igh )
leads o mul iple wo ms being used wi hin he same connec ed
componen . In o de o ob ain ins ance segmen a ion om
such images, manual cu a ion o pos -p ocessing, such as
wa e shed, is necessa y o sepa a e he connec ed objec s in o
di e en ins ances.
Finally i is also impo an o know ha ained andom o es
classi ie canno easily be ained on se s o e y di e se images.
Deep lea ning app oaches such as Cellpose (S inge e al., 2021),
in con as , show much g ea e po en ial o gene alize well e en
when ained on a la ge and di e se body o mic oscopy da a.
None heless, LABKIT can be used o segmen a wide ange o
images as and a high quali y. This is ue as long as objec s a e
isibly sepa a ed om one ano he and can be dis inguished by
he il e esponses LABKIT compu es pe pixel.
5. SOFTWARE AND WORKFLOW
INTEGRATION
LABKIT’s au oma ic segmen a ion is no limi ed o he da ase
i was ained on. Because he ained classi ie can be sa ed
o la e use, i can be applied o simila new images. While
ensu ing ep oducibili y o he esul s, i also helps main aining
consis ency in he image segmen a ion. Manually loading bo h
images and ained classi ie in LABKIT o mul iple se s o
images is a epe i i e ask ill-sui ed o an au oma ed wo k low.
The e o e, o simpli y he in eg a ion in o exis ing wo k lows
in Fiji, LABKIT can be easily called om he ImageJ mac o
language. Fo ins ance, a simple mac o sc ip can open mul iple
da ase s and segmen each o hem using a ained classi ie .
Image segmen a ion can be u he accele a ed by unning
he p ocess on GPUs hanks o CLIJ (Haase e al., 2020).
Once CLIJ is p ope ly se up, GPU accele a ion is a ailable
o LABKIT in bo h g aphical in e ace and mac o commands.
GPU p ocessing is pa icula ly bene icial in he case o la ge
images, o which i allows sho ening he leng hy segmen a ion
asks. Pe o ming GPU-accele a ed segmen a ion in LABKIT is a
ma e o ac i a ing a checkbox, and does no p esen addi ional
complexi y o use s.
Some images, howe e , a e a oo la ge o be p ocessed
on a consume machine in a easonable amoun o ime,
i hey can be s o ed a all on such a compu e . Fo such
da a, mode n wo k lows eso o he use o HPC clus e s,
which a e pu posely buil o high compu ing pe o mances
wi h la ge a ailable memo y. LABKIT o e s a command line
ool (A z , 2021a) allowing ad anced use s o segmen images on
HPC clus e s.
The capabili y o ex ending LABKIT and e-using
i s componen s is illus a ed by in eg a ion wi h he
comme cial Ima is so wa e (Ox o d Ins umen s, UK) ia
he ecen ly eleased ImgLib2-Ima is compa ibili y b idge.
In his con ex , LABKIT ope a es di ec ly on da ase s ha
a e anspa en ly sha ed (wi hou duplica ion) be ween
Ima is and ImgLib2 (Pie zsch e al., 2012). These da ase s
can be a bi a ily la ge, as bo h Ima is and ImgLib2
implemen sophis ica ed caching schemes. In he same
ashion, ou pu segmen a ion masks a e anspa en ly
sha ed wi h he unning Ima is applica ion, making
addi ional ile impo /expo s eps unnecessa y. Impo an ly,
his unc ionali y can also be igge ed and con olled
di ec ly om Ima is o in eg a e i in o s eamlined objec
segmen a ion wo k lows.
6. PERFORMANCE OF LABKIT
In o de o p ocess la ge images on consume compu e s,
so wa e packages mus be able o load he da a in memo y,
p ocess i and sa e he esul s, all wi hin he cons ain s o
he machine. In LABKIT, his is achie ed by eading only he
po ions o he image ha a e displayed o he use , hanks
o he use o he HDF5 o ma (Folk e al., 2011) and he
BigDa aViewe (Pie zsch e al., 2015). The image is u he
p ocessed in chunks using ImgLib2 (Pie zsch e al., 2012).
As a esul , LABKIT is capable o p ocessing a bi a ily la ge
F on ie s in Compu e Science | www. on ie sin.o g 7Feb ua y 2022 | Volume 4 | A icle 777728
A z e al. LABKIT: Labeling and Segmen a ion Toolki
images and is compa ible wi h GPU accele a ion and dis ibu ed
compu a ion on HPC clus e s.
To illus a e his, we segmen ed a 13.4 gigapixel image (482 x
935 x 495 x 60 pixels, 25 GB) on a single lap op compu e , wi h
and wi hou GPU, and wi h di e en nodes o an HPC clus e
(see Table 1). The image was ex ac ed and 2x down-sampled
om he Fluo-N3DL-TRIF da ase made a ailable o he Cell
T acking Challenge (Maška e al., 2014; Ulman e al., 2017; Jain
e al., 2020) benchma k compe i ion. Running he segmen a ion
on he lap op using GPU accele a ion sped up he compu a ion
by 7.5 old, illus a ing he bene i o ha nessing GPU powe
o p ocessing la ge images. While unning compu a ion on an
HPC clus e comes wi h o e head, inc easing he numbe o CPU
nodes sho ens he compu a ion d ama ically, eaching a 40-
old imp o emen om 1 CPU node o 50. Finally, GPU nodes
on an HPC allow o mo e pa alleliza ion o he compu a ion
and he e o e e en highe compu a ional speed-up on he
segmen a ion ask, wi h 10 GPU nodes p ocessing he da a in
sligh ly o e a minu e.
Fu he mo e, we ained and op imized a classi ie on he
Fluo-N3DL-TRIF da ase (o iginal sampling), he la ges da ase
o he Cell T acking Challenge ( aining da ase o size 320
GB, e alua ion da ase o size 467 GB), and submi ed i o
e alua ion agains undisclosed g ound- u h. The segmen a ion
o bo h aining and e alua ion da ase s was pe o med on an
HPC clus e . LABKIT pixel classi ica ion anked as he highes
pe o ming segmen a ion me hod on his da ase o all h ee
e alua ion me ics (OPCSB,SEG and DET) (CTC, 2021). Mo e
speci ically, LABKIT segmen a ion ob ained he ollowing sco es:
OPCSB =0.895 (0.886 o he second highes sco ing en y),
SEG =0.793 (0.776) and DET =0.997 (0.997), pe o ming
be e han he o he en ies, including classical (bandpass
segmen a ion) o deep lea ning (con olu ion neu al ne wo k)
algo i hms. As opposed o he deep lea ning algo i hm o which i
was compa ed, Labki only used a ew hund ed pixels as g ound-
u h, dis ibu ed h oughou a small ac ion o he aining
da ase (7 ames). Finally, LABKIT’s classi ie was simply ained
h ough he LABKIT g aphical in e ace, illus a ing i s ease
o use.
7. DISCUSSION AND CONCLUSION
LABKIT is a labeling so wa e ool designed o be in ui i e
and simple o use. I ea u es a obus pixel classi ica ion
algo i hm aimed a segmen ing images be ween mul iple classes
wi h e y li le manual labeling equi ed. Simila o o he
ools o he BigDa aViewe amily (Pie zsch e al., 2015;
Wol e al., 2018; Hö l e al., 2019; Tische e al., 2020),
i in eg a es seamlessly in o he SciJa a and Fiji ecosys em.
I can be easily ins alled h ough Fiji and inco po a ed in o
es ablished wo k lows using ImageJ’s mac o language. The esul s
o LABKIT’s segmen a ion can be u he analyzed in Fiji
o expo ed o o he so wa e pla o ms, such as CellP o ile
(McQuin e al., 2018), QuPa h (Bankhead e al., 2017), o
Ilas ik (Be g e al., 2019).
Manual labeling, in bo h 2D and 3D, is also made
easy by LABKIT. O he al e na i es exis , among which
QuPa h (Bankhead e al., 2017) (2D), napa i (napa i con ibu o s,
2019) o Pain e a (Lei e e al., 2021). In pa icula , Pain e a is
speci ically ailo ed o 3D labeling o c owded en i onmen , bu
a he cos o a s eepe lea ning cu e.
LABKIT is compa ible wi h a wide ange o image o ma s
since image da a can be loaded di ec ly om Fiji using Bio-
Fo ma s (Linke e al., 2010). None heless, in o de o ully
bene i om LABKIT op imiza ions o la ge images, use s
mus i s con e hei e aby e-sized images o a ile o ma
allowing high-speed access o a bi a y loca ed sub- egions o he
image. This s a egy is also employed by o he so wa e, wi h
he example o Ilas ik (Be g e al., 2019). One such o ma is
HDF5 (Folk e al., 2011), and LABKIT uses in pa icula he
BigDa aViewe HDF5+XML a ian . In Fiji, images can easily
be sa ed in his o ma using BigS i che (Hö l e al., 2019)
o Mul i iew-Recons uc ion (P eibisch e al., 2014; Icha e al.,
2016).
In he Cell T acking Challenge (Ulman e al., 2017; CTC,
2021), LABKIT segmen a ion ou pe o med o he en ies on a
pa icula da ase , one being a deep lea ning app oach. This
me hod was designed as pa o a cell segmen a ion and acking
pipeline on a ious images, and i is likely ha ecen and
specialized deep lea ning segmen a ion algo i hms, such as
S a Dis (Schmid e al., 2018) o CellPose (S inge e al., 2021),
would pe o m o e all be e (Bal issen e al., 2018; Mo one
e al., 2020). Ye , he ull po en ial o deep lea ning algo i hms
is only eached when a su icien amoun o g ound- u h da a is
a ailable, which is oo equen ly he limi ing ac o . Gene a ing
g ound- u h da a o a deep lea ning me hod is a edious
endea o wi hou he insu ance o a pe ec segmen a ion esul .
A sa e s a egy is he e o e o i s y shallow lea ning o
segmen a ion asks, be o e e en hinking o mo ing o deep
lea ning algo i hms. In cases whe e highe segmen a ion quali y
is uly necessa y, cu a ed esul s om shallow lea ning can be
used o gene a e he massi e amoun o g ound- u h equi ed
o ain a deep lea ning algo i hm. As seen p e iously, LABKIT
is use ul in all hese scena ios since i can be used o manually
gene a e g ound- u h anno a ions o o segmen he images
wi h shallow lea ning be o e cu a ing he esul s in o de o use
hem as g ound- u h o o he lea ning-based algo i hms (see
Figu e 6).
In he u u e, we in end o ex end LABKIT’s unc ionali ies
o imp o e manual and au oma ed segmen a ion. Fo ins ance,
we will add a magic wand ool o selec , ill, use o dele e labels
based on he pixel classi ica ion. Fu he mo e, we aim o add new
segmen a ion algo i hms, such as he deep lea ning algo i hm
DenoiSeg (Buchholz e al., 2020) al eady a ailable in Fiji. In
ecen yea s, no el in e ac i e deep lea ning app oaches ha e also
been shown o educe he need o la ge amoun s o densely
labeled g ound- u h da a. In gene al, hese app oaches combine
deep lea ning wi h in e ac i e use guidance, o ins ance clicks
on he ex eme poin s o objec s (Maninis e al., 2018), inside-
ou side guidance (Zhang e al., 2020), clicks wi hin objec s and
bounda ies ha a e i e a i ely e ined (Luo e al., 2021) and
a combina ion o clicks and squiggles inside objec s (Alemi
F on ie s in Compu e Science | www. on ie sin.o g 8Feb ua y 2022 | Volume 4 | A icle 777728
A z e al. LABKIT: Labeling and Segmen a ion Toolki
FIGURE 6 | LABKIT’s i e a i e and in e ac i e segmen a ion used o g ound- u h gene a ion: manual labeling, au oma ic segmen a ion and cu a ion in LABKIT enable
easy and apid image segmen a ion, whose esul s can be u he p ocessed o used as g ound- u h o deep-lea ning classi ie s.
Koohbanani e al., 2020). Howe e , hese app oaches a e no
in widesp ead use in bio-image analysis and o he mos pa
implemen ed in Py hon. LABKIT could po en ially se e as an
easy- o-use pla o m o such me hods by implemen ing hei
labeling s a egies in he use in e ace and in e acing hei
amewo k wi h Ja a, he eby making hem widely accessible
o he biomedical communi y. LABKIT sou ce code is open
sou ce and can be ound online (A z , 2021b), oge he
wi h i s command-line in e ace (A z , 2021a), u o ials and
documen a ion (A z , 2021c).
8. METHODS
8.1. Timing Ins ance Segmen a ion
Gene a ion
The da ase consis ed o all 256 x 256 images (N= 26) in
he es sample o S a Dis (Schmid e al., 2018), o iginally
published as pa o he 2018 Da a Science Bowl (Caicedo e al.,
2019) (subse o s age1_ ain, accession numbe BBBC038, B oad
Bioimage Benchma k Collec ion). The images we e loaded in
LABKIT as a s ack and spa sely labeled (sc ibbles). A classi ie
was hen ained wi h he de aul il e se ings: "o iginal image,"
"Gaussian blu ," "di e ence o Gaussians," "Gaussian g adien
magni ude," "Laplacian o Gaussian," and "Hessian eigen alues,"
wi h sigma alues: 1, 2, 4, and 8. The esul s we e sa ed
and hen manually cu a ed using he b ush and e ase ools.
Finally, he same o iginal image s ack was densely manually
labeled a esh. The o al ime equi ed o p ocess all images
was measu ed using a ch onome e o i) LABKIT au oma ed
segmen a ion, including he spa se manual labeling, ii) he
p e ious s ep ollowed by a cu a ion s ep and iii) dense manual
labeling. In o de o e alua e he segmen ed images, connec ed
componen s we e compu ed (4-connec i i y) and gi en unique
pixel alues (ins ance segmen a ion). Quali y me ics sco es
we e calcula ed as he a e age p ecision wi h h eshold 0.5 as
de ined in S a Dis (Schmid e al., 2018). We used dense manual
labeling pe o med by ano he obse e as e e ence images, and
compu ed he me ics sco e o he esul s ob ained in i), ii),
and iii). The a e age me ics o e he images we e calcula ed as
a weigh ed a e age o each indi idual image, whe e he weigh s
we e he numbe o ins ances in he e e ence image.
8.2. Speed Benchma k
The da ase was downloaded om he Cell T acking
Challenge (Ulman e al., 2017) websi e, and consis ed o
he i s aining da ase o he Fluo-N3DL-TRIF example. The
da ase was down-sampled by a ac o 2 in o de o educe i s size
and simpli y he benchma king. The da ase was hen sa ed in
he BigDa aViewe XML+HDF5 o ma using BigS i che (Hö l
e al., 2019). LABKIT was used o d aw a ew sc ibbles on bo h
backg ound and nuclei a eas, and o ain a andom o es
classi ie using he de aul se ings. The ained model was hen
sa ed. The LABKIT command line ool was used o un he
benchma k expe imen on a Dell XPS 15 lap op (32 MB RAM,
In el Co e i7-6700HQ CPU wi h 8 co es, GeFo ce GTX 960M
GPU) and on an HPC clus e , wi h bo h CPU (256 GB RAM,
In el Xeon CPU E5-2680 3 wi h 2.5 GHz and 24 co es) and
GPU (512 GB RAM, In el Xeon CPU E5-2698 4 wi h 2.2 GHz
and 40 co es, wi h wo GeFo ce GTX 1080 GPUs) nodes. The
segmen a ion esul s on he HPC we e sa ed in he N5 (Saal eld,
2017) o ma o maximize w i ing speed. Benchma king included
ead/w i e o image da a o m disc, op ional da a ans e o
he GPU, compu a ion o ea u e images and classi ica ion
all oge he .
F on ie s in Compu e Science | www. on ie sin.o g 9Feb ua y 2022 | Volume 4 | A icle 777728