Characterisation of sediment patterns and benthic megafauna distribution using automated underwater image analysis
Abstract
Seafloor habitat classification and marine biodiversity assessment studies are core fundamental activities in marine science research. These studies are crucial because they provide foundational data for the establishment of accurate and reliable baseline marine information. Established marine biodiversity and habitat baselines provide a sound scientific basis for tracking changes in ecosystem health, thereby supporting evidence-based decision making for the overall protection of marine environments. Mapping these vast remote marine ecosystems is typically achieved using (medium resolution) acoustics methods, whereas ground truthing and detailed investigations of specific target sites are usually performed using high resolution optical imaging. While marine scientists have used images to study marine ecosystems for decades, the manual annotation approaches that were traditionally used to interpret the images are no longer feasible in this terrabyte-scale marine big data regime. To complement conventional manual approaches, automated image analysis workflows are required to e.g improve visibility of degraded raw images, as well as to expedite the transformation of these terabyte-scale seafloor images (and videos) into semantic habitat classes or megafaunal taxa. The workflows should also investigate and account for potential sampling and scaling biases that arise from e.g the failure of the imaging platform to maintain a consistent altitude above the seafloor. Finally, the workflows must also allow for seamless integration of the generated annotations with spatio-ecological models, in order to provide geographic context to the image-derived annotations. Therefore, this thesis implements integrated workflows centered around the above mentioned aspects, and reports detailed findings based on specific case studies from scientific expeditions to the Clarion-Clipperton Zone in the Pacific, as well as the tropical North Atlantic.
Full text
CHARACTERISATION OF SEDIMENT PATTERNS AND BENTHIC MEGAFAUNA
DISTRIBUTION USING AUTOMATED UNDERWATER IMAGE ANALYSIS
DISSERTATION
in ulllmen o he equi emen s o he deg ee o
Doc o e um na u alium (D . e . na )
o he Facul y o Ma hema ics and Na u al Sciences
a he Ch is ian-Alb ech s-Uni e si ä zu Kiel
Submi ed by
BENSON ONYANGO MBANI
GEOMAR Helmhol z Cen e o Ocean Resea ch Kiel
Kiel, Oc obe 2024
1
Examine s:
1. P o . D . Jens G eine
2. P o . D .-Ing. Reinha d Koch
Da e o dispu a ion: 25.10.2024
2
“How we spend ou days is, o cou se, how we spend ou li es”
Annie Dilla d
3
Abs ac
Seaoo habi a classica ion and ma ine biodi e si y assessmen s udies a e co e undamen al
ac i i ies in ma ine science esea ch. These s udies a e c ucial because hey p o ide ounda ional
da a o he es ablishmen o accu a e and eliable baseline ma ine in o ma ion. Es ablished
ma ine biodi e si y and habi a baselines p o ide a sound scien ic basis o acking changes in
ecosys em heal h, he eby suppo ing e idence-based decision making o he o e all p o ec ion
o ma ine en i onmen s. Mapping hese as emo e ma ine ecosys ems is ypically achie ed
using (medium esolu ion) acous ics me hods, whe eas g ound u hing and de ailed
in es iga ions o specic a ge si es a e usually pe o med using high esolu ion op ical
imaging. Recen echnological ad ancemen s in imaging senso s and s o age memo y ha e
enabled he acquisi ion o huge olumes o seaoo images e en om a single scien ic
expedi ion. While ma ine scien is s ha e used images o s udy ma ine ecosys ems o decades,
he manual anno a ion app oaches ha we e adi ionally used o in e p e he images a e no
longe easible in his ma ine big da a egime. To complemen hese manual app oaches,
au oma ed image analysis wo kows a e equi ed o no only imp o e he isibili y o deg aded
aw images, bu o also expedi e he ans o ma ion o hese e aby e-scale seaoo images (and
ideos) in o seman ic habi a classes o mega aunal axa. The wo kows should also implemen
me hodologies o in es iga ing po en ial sampling and scaling biases e.g as a esul o a ying
al i ude (o speed) o he imaging pla o m, as well modules o co ec ing a i ac s in oduced
by hese biases. These co ec ions in ol e e.g he s anda diza ion o isual oo p in s ha a y
depending on espec i e image acquisi ion heigh s, o he es ablishmen o xed-size sampling
uni s wi hin which o con e absolu e mega aunal coun s in o abundances ha a e no malized
ela i e o ac ual obse ed seaoo a ea. Finally, he wo kows mus also allow o seamless
in eg a ion o he gene a ed anno a ions wi h spa io-ecological models. This in eg a ion is
c ucial because i p o ides geog aphic con ex o he image-de i ed anno a ions, which in u n
enables comp ehensi e cha ac e isa ions o mul i-scale spa ial dis ibu ion pa e ns o habi a s
and biodi e si y. The in eg a ion also allows o an objec i e assessmen o en i onmen al
d i e s (e.g ba hyme y, empe a u e, salini y, e ce e a) ha inuence he obse ed dis ibu ion
pa e ns. The e o e, his hesis implemen s in eg a ed wo kows cen e ed a ound he abo e
men ioned aspec s, and epo s de ailed ndings based on specic case s udies om scien ic
expedi ions o he Cla ion-Clippe on Zone in he Pacic, as well as he opical No h
A lan ic.
5
Zusammen assung
S udien zu Klassizie ung on Lebens äumen am Mee esboden und zu Bewe ung de
ma inen Biodi e si ä sind g undlegende Ke nak i i ä en de mee eswissenscha lichen
Fo schung. Diese S udien sind on en scheidende Bedeu ung, da sie g undlegende Da en ü
die E s ellung genaue und zu e lässige Basisin o ma ionen übe die Mee e lie e n. Fes geleg e
Basisda en zu ma inen Biodi e si ä und zu Lebens äumen bie en eine solide wissenscha liche
G undlage ü die Ve olgung on Ve ände ungen de Ökosys emgesundhei und un e s ü zen
so eine e idenzbasie e En scheidungsndung zum allgemeinen Schu z de Mee esumwel . Die
Ka ie ung diese iesigen abgelegenen Mee esökosys eme e olg in de Regel mi hil e on
(mi le e Auösung) akus ischen Me hoden, wäh end die Bodenwah nehmung und
de aillie e Un e suchungen bes imm e Ziels ando e no male weise mi hil e hochauösende
op ische Bildgebung du chge üh we den. Jüngs e echnologische Fo sch i e bei
Bildsenso en und Speiche medien haben die E assung iesige Mengen on
Mee esbodenbilde n soga on eine einzigen wissenscha lichen Expedi ion e möglich .
Wäh end Mee eswissenscha le sei Jah zehn en Bilde zu Un e suchung ma ine Ökosys eme
e wenden, sind die manuellen Anno a ionsme hoden, die adi ionell zu In e p e a ion de
Bilde e wende wu den, in diesem ma inen Big-Da a-Regime nich meh du ch üh ba . Als
E gänzung zu diesen manuellen Ansä zen sind au oma isie e Bildanalyse-Wo kows
e o de lich, die nich nu die Sich ba kei on Rohbilde n mi schlech e Quali ä e besse n,
sonde n auch die Umwandlung diese Mee esbodenbilde (und - ideos) im Te aby e-Maßs ab
in seman ische Habi a klassen ode Mega auna-Taxa beschleunigen. Die Wo kows soll en auch
Me hoden zu Un e suchung po enzielle S ichp oben- und Skalie ungs e ze ungen
implemen ie en, die beispielsweise du ch un e schiedliche Höhen (ode Geschwindigkei en) de
Bildgebungspla o m en s ehen, sowie Module zu Ko ek u on A e ak en, die du ch diese
Ve ze ungen e u sach we den. Diese Ko ek u en um assen beispielsweise die
S anda disie ung isuelle Fußabd ücke, die je nach den jeweiligen Bildau nahmehöhen
a iie en, ode die Ein ich ung on S ichp obeneinhei en mi es e G öße, inne halb de e
absolu e Mega auna-Zählungen in Häugkei en umgewandel we den können, die ela i zu
a sächlich beobach e en Mee esbodenäche no malisie sind. Schließlich müssen die
Wo kows auch eine nah lose In eg a ion de gene ie en Anme kungen in
äumlich-ökologische Modelle e möglichen. Diese In eg a ion is on en scheidende
Bedeu ung, da sie den aus den Bilde n abgelei e en Anme kungen einen geog aschen Kon ex
e leih , was wiede um eine um assende Cha ak e isie ung meh skalige äumliche
Ve eilungsmus e on Lebens äumen und A en iel al e möglich . Die In eg a ion e möglich
auch eine objek i e Bewe ung on Umwel ak o en (z. B. Ba hyme ie, Tempe a u , Salzgehal
usw.), die die beobach e en Ve eilungsmus e beeinussen. Dahe implemen ie diese A bei
in eg ie e A bei sabläu e, die sich au die oben genann en Aspek e konzen ie en, und
6
be ich e übe de aillie e E gebnisse basie end au spezischen Falls udien on
wissenscha lichen Expedi ionen in die Cla ion-Clippe on-Zone im Pazik sowie in den
opischen No da lan ik.
7
Lis o Abb e ia ions
A.I A icial In elligence
ADCP Acous ic Dopple Cu en P ole
ANOSIM Analysis o simila i ies
AUV Au onomous Unde wa e Vehicle
BIIGLE Bio-Image Indexing and G aphical Labeling En i onmen
CCZ Cla ion-Clippe on Zone
CLAHE Con as Limi ed Adap i e His og am Equaliza ion
CTD Conduc i i y, Tempe a u e, and Dep h
GPU G aphics P ocessing Uni
LED Ligh Emi ing Diode
LISA Local Indica o s o Spa ial Associa ion
nm-MDS Non-me ic Mul idimensional Scaling
OFOS Ocean Floo Obse a ion Sys em
PCA P incipal Componen s Analysis
POC Pa icula e O ganic Ca bon
ROV Remo ely Ope a ed Vehicle
R-CNN Region-based Con olu ional Neu al Ne wo ks
RV Resea ch Vessel
SIMPER Simila i y Pe cen ages
USBL Ul a Sho Baseline
XOFOS Ex ended Ocean Floo Obse a ion Sys em
8
(F ees one e al., 2011). The decline in ma ine biodi e si y could also be a ibu ed o
an h opogenic ac o s such as bo om awling, pollu ion, in oduc ion o in asi e species, and
human-induced clima e change which aises ocean empe a u es and causes acidica ion (Sala
and Knowl on, 2006) To be e quan i y and add ess his biodi e si y decline, globally
coo dina ed eo s a e equi ed o suppo an inc ease in no only he numbe o scien ic
expedi ions, bu also spa ial- empo al ex en s co e ed by espec i e sampling
campaigns(Canonico e al., 2019). Specically o he deep sea, some o hese coo dina ed
ini ia i es should be di ec ed owa ds he de elopmen o acous ic and op ical imaging
pla o ms ha allow o de ailed non-in asi e in es iga ions o emo e ben hic ecosys ems
(Le in e al., 2019). To maximize he esou ce ulness o he acqui ed images, ideos, and
auxilia y senso eadings, co esponding in es men s a e also equi ed o incen i ise inno a ion
in ma ine da a science wo kows (Guidi e al., 2020). These inno a ions would signican ly
expedi e he p ocess o ans o ming he huge olumes o (po en ially uns uc u ed) aw ma ine
da ase s in o ac ionable insigh s, which can be used o in o m policy decisions e.g ega ding
delinea ion o ma ine p o ec ed a eas. Addi ional echnological in es men s should also be
di ec ed owa ds upg ading compu a ional in as uc u es such as ede a ed da a po als and
GPU clus e s (Guidi e al., 2020).
Recen ad ances in da a science me hods ha e demons a ed ema kable capabili y o he
ecien ex ac ion o seman ic anno a ions om e aby e-scale unde wa e images and ideos
(Mbani e al., 2023a, 2022a). S a e-o - he a compu e ision models ha ha e been p e ained
on la ge benchma k da ase s (Deng e al., 2009) can now be eely downloaded om
eposi o ies such as Tenso Flow Hub, PyTo ch Hub and HuggingFace Model Hub (Pang e al.,
2020) (Paszke e al., 2019) (Jain, 2022). Downloaded models can be s aigh o wa dly deployed
as-is o pe o m common image analysis asks such as isibili y enhancemen , classica ion,
segmen a ion, objec de ec ion and acking. Conside ing ha p e ained models we e
o iginally ained using a la ge se o e es ial images con aining common objec s (Shin e al.,
2016), he models become e y eec i e a e ne- uning agains ma ine-specic images (Luo e
al., 2019). This is because p e aining al eady exposes he models o di e se abs ac ea u es and
pa e ns in e es ial images, causing he ne- uning o quickly con e ge a e adap ing he
model o he unique p ope ies o he ma ine en i onmen (Pede sen e al., 2019). Howe e , he
ne- uning p ocess s ill equi es a subs an ial amoun o example images ha a e each labeled
(e.g wi h a co esponding habi a class o mega aunal axa). This equi emen p o es
challenging in ma ine sciences, in pa because anno a ing images a e e e y expedi ion is cos ly
and unscalable (Mbani e al., 2023a). Besides, ma ine en i onmen s (especially in he deep sea)
na u ally exhibi pa e ns o educing mega aunal abundance wi h dep h (Haed ich e al., 1980)
(Saeedi e al., 2022). As a esul , he majo i y o acqui ed images do no con ain isible
mega auna, and e en hose wi h mega auna a e ypically unknown o anno a o s ahead o ime
(Mbani e al., 2023a). As a esul , inspec ing he en i e image da ase sequen ially jus o nd
15
he ew wi h isible mega auna o be anno a ed is no only labo ious, bu also undesi able. This
challenge calls o he de elopmen o au oma ed anno a ion wo kows ha oe high
accu acy, eciency, epea abili y, and ha can scale o e en bigge -sized ma ine da ase s. The
au oma ed wo kows mus also be capable o in eg a ing wi h spa io-ecological models o
geog aphically con ex ualize he image-de i ed anno a ions.
Figu e 2: Illus a ion o he clea upwa d end in he pe o mance o compu e ision
models ela i e o he human pe o mance baseline. Fo cla i y, he gu e only shows baseline
compa isons o image classica ion (imageNe Top-5) models. Da a o pe o mance
benchma ks ob ained om he AI Index Repo 2024 (Maslej e al., 2024)
The dis ibu ion o ben hic mega auna in geog aphic space is nei he andom no uni o m
(Legend e and Fo in, 1989). Ins ead, bio ically-simila mega auna gene ally exhibi geog aphic
clus e ing pa e ns ha a e in u n inuenced by a iabili y in en i onmen al ac o s such as
ba hyme y, geomo phology and empe a u e g adien s, as well as ecological p ocesses such as
POC ux, p eda o -p ey in e ac ions, b eeding pa e ns e ce e a (Lacha i é and Me axas, 2018).
Accu a e cha ac e iza ion o hese mega aunal dis ibu ion pa e ns is he e o e equi ed in
16
o de o de elop undamen al baseline da a o moni o ing he heal h o ma ine ecosys ems
(Ha is and Bake , 2012), as well as o de eloping p edic i e models ha simula e e.g po en ial
en i onmen al impac s o deep sea polyme allic Mn-nodule mining (Peuke e al., 2018).
The e o e, his hesis builds upon he o egoing ounda ional concep ual amewo ks. The
o e a ching goal o he hesis is o de elop a se o au oma ed da a science wo kows ha a e
capable o quick bu objec i e seman ic anno a ion o ben hic habi a ypes and mega aunal
axa om la ge sequences o high- esolu ion op ical images. The gene a ed anno a ions a e hen
used downs eam o cha ac e ize ne- and egional-scale spa io-ecological dis ibu ion pa e ns.
Examples a e p o ided based on case s udies om deep seabed a eas o he Pacic and opical
No h A lan ic.
1.1 Mo i a ion and Objec i es
1.1.1 Ma ine Science Pe spec i e
Technological ad ances in ma ine sciences a e d i ing up demand o ma ine da a science
expe ise (Guidi e al., 2020). In pa icula , unde wa e imaging pla o ms a e nowadays
equipped wi h mul iple senso s ha enable de ailed in es iga ions o la ge ex en s o ma ine
ecosys ems (Pu se e al., 2019). These su eys ypically gene a e huge olumes o
high- esolu ion seaoo images (and ideos) ha equi e bo h s eamlined da a managemen
p o ocols (Schoening e al., 2018), as well as au oma ed da a analysis wo kows ha a e buil
upon eme ging digi al echnologies such as da a science (Guidi e al., 2020). Examples o hese
ma ine echnological ad ances include e he ed pla o ms such as he Ocean Floo Obse a ion
Sys ems (OFOS) (Pu se e al., 2019) and Remo ely Ope a ed Vehicles (ROVs) (Hu enne e al.,
2018). These e he ed pla o ms a e ypically connec ed o he ship using an umbilical be
op ic cable ha deli e s bo h powe and p opulsion o he pla o m, while simul aneously
ansmi ing li e ideo eed back o he compu e on he ship o eal ime anno a ion using e.g
he OFOP so wa e. The OFOS is usually owed behind he ship along p edened waypoin s,
which makes i sui able o isual in es iga ions o he seabed along linea ansec s (Mbani e
al., 2022a). In con as , ROVs a e ypically pilo ed and he e o e possess he maneu e abili y o
in es iga e complex ben hic e ains, including specic si es o in e es as deemed by he
ma ine scien is (du ing he su ey). Non- e he ed imaging pla o ms such as Au onomous
Unde wa e Vehicles (AUVs) ope a e independen o di ec con ol by an ope a o , and can
he e o e be p og ammed o sel -comple e p e planned missions o su eying la ge a eas o
ma ine ecosys ems using bo h acous ics and op ical imaging modali ies (Hu enne e al., 2018).
This au onomy o AUVs allows o he sampling ope a ions o p oceed in pa allel, which
maximizes ship ime, widens he ange o ma ine obse a ions, and inc eases he p oduc i i y o
17
he expedi ion o e all. In addi ion, he acqui ed ma ine da ase s can also be geo e e enced
because he imaging pla o ms also eco d ime-indexed na iga ion in o ma ion based on ei he
acous ic ansponde s (Heg enæs e al., 2009) o ul asho baselines (USBL) (Rigby e al.,
2006). Depending on he specic ma ine science ques ion unde in es iga ion, he pla o ms
can ei he be deployed ad-hoc, o as ne wo ks o in e connec ed long- e m obse a ion s a ions
(Wang e al., 2022). The abo e ma ine echnological ad ances gene a e ma ine big da ase s ha
a e no mally cu a ed and a chi ed on ede a ed da a po als such as PANGAEA(Felden e al.,
2023) and BIIGLE (Langenkämpe e al., 2017). The e o e, he need o ecien ly ans o m
hese ma ine big da ase s in o ac ionable insigh s is a s ong mo i a ion o in es ing in ma ine
da a science.
Figu e 3: Ex ended Ocean Floo Obse a ion Sys em (XOFOS) being lowe ed in o he wa e
column o su ey deep seabed a eas o he opical No h A lan ic du ing c uise M182.
Ma ine scien is s need o pe o m ou ine cha ac e iza ion o habi a s and biodi e si y because
hese a e key indica o s o a heal hy and unc ioning ecosys em (Galpa so o e al., 2014). To
18
eec i ely pe o m he cha ac e iza ions, me hods a e equi ed o con e quali a i e isual
in o ma ion (in seaoo images and ideos) in o objec s ha ma ine esea che s can manipula e
compu a ionally e.g h ough s a is ical and p ocess-based modeling (Du den e al., 2016). This
is a s ong mo i a ion o adop ing da a science echniques such as mul i-scale ea u e
ex ac ion (Lu e al., 2023), which allow o a s aigh o wa d encoding o abs ac isual
in o ma ion in o compac ec o ep esen a ions ha can be manipula ed compu a ionally. Fo
example, hand-c a ed ex u e and en opy ea u es a e capable o ep esen ing he dis ibu ion
o Mn-nodules ha appea on seaoo images as da k nea -ci cula pa ches (Mbani e al.,
2022a). On he o he hand, p e- ained con olu ional neu al ne wo ks such as Incep ion V3
ha e shown ema kable capabili ies o encoding egions o he seabed exhibi ing sub le
a iabili y in subs a e composi ion (Szegedy e al., 2015). Ma ine scien is s a e also in e es ed
in iden i ying mic ohabi a s ha a e nes ed wi hin he la ge dominan habi a s e.g o
unde s and p ocesses ha d i e habi a selec ion decisions by species (Vaudo and Hei haus,
2013). These mic ohabi a s can be easily de ec ed h ough da a science me hodologies o
anomaly and no el y de ec ion e.g he isola ion Fo es (iFo es ) algo i hm (Liu e al., 2008).
Ma ine scien is s also need high- esolu ion habi a maps o accu a ely pa ame e ize ecological
models, since hese maps allow modele s o a oid making inco ec assump ions ega ding
andom o uni o m spa ial dis ibu ion o subs a e cha ac e is ics (E ans e al., 2015). These
habi a maps can be accu a ely p oduced using supe ised machine lea ning echniques like
deep lea ning, which ha e so a demons a ed capaci y o accu a ely de e mine bo h linea and
nonlinea decision bounda ies o classica ion (p o ided he e a e sucien labeled aining
examples) (Shin e al., 2016). E en in he absence o labeled anno a ions, unsupe ised me hods
such as K-means ha e p o en use ul o quick p elimina y so ing o images based on na u al
g oupings, a e which seman ic habi a classes can be manually assigned la e (Mbani e al.,
2022a). Ben hic biodi e si y assessmen s a e also equi ed by ma ine scien is s o unde s and
he s a us o ma ine ecosys em unc ioning and se ices (Galpa so o e al., 2014) . To his end,
da a science models such as Fas e R-CNN (Ren e al., 2015) and YOLO (Du, 2018) ha e
demons a ed capaci y o au oma ed de ec ion, localiza ion and coun ing o o ganisms om
images and ideos wi h high accu acy. Finally, he e is a need by ma ine scien is s o o mula e
hypo heses explaining he inuence o en i onmen al a iables on he obse ed spa ial
dis ibu ion pa e ns o habi a s and biodi e si y (Legend e and Fo in, 1989). These needs can
be add essed h ough dimensionali y educ ion echniques in da a science (e.g p incipal
componen s analysis and non-me ic mul idimensional scaling) (Bakke , 2024).
The ac ha classical ma ine science wo kows al eady employ mul i a ia e s a is ics o
modeling phenomena simplies in eg a ion wi h da a science me hods (also deeply oo ed in
s a is ical o mula ion) (“Communi y ecology in he age o mul i a ia e mul iscale spa ial
analysis - D ay - 2012 - Ecological Monog aphs - Wiley Online Lib a y,” n.d.). Fo example,
ma ine scien is s ypically map ou s a is ically signican ho spo s o mega aunal abundances
19
in ne-scale using spa ial au oco ela ion echniques (local Mo ans’ I), which is a special case o
au oco ela ion in s a is ics. Also, ma ine scien is s ypically assess in e ela ionships be ween
axa and en i onmen al ac o s by p ojec ing he a iables on o a wo-dimensional o dina ion
ea u e space e.g using non-me ic mul idimensional scaling (Bakke , 2024); hese o dina ion
echniques a e a special case o dimensionali y educ ion in da a science. This seamless
in eg a ion allows ma ine esea che s o app op ia ely le e age he s eng hs o ei he ma ine
science and da a science wo kows whene e app op ia e. Specically, pa e n ecogni ion
capabili ies o da a science models can be used o au oma e ime-consuming e o -p one asks
like cleaning, missing alue impu a ion, and seman ic anno a ion o po en ially uns uc u ed
ma ine da ase s. On he o he hand, spa io-ecological models ha a e ypically o mula ed
a ound mechanis ic simula ion o ecological p ocesses (and ha e been es ed o e se e al
decades) can be used o p o ide nuanced in e p e a ions o obse ed pa e ns (Cudding on e
al., 2013). Beyond anno a ions, ma ine scien is s can use (aspec s o ) da a science o mula ions
as an al e na i e o some spa io-ecological models ha by de aul make un ealis ic s a is ical
assump ions o no mali y among inpu a iables (Legend e and Fo in, 1989). In such
si ua ions, algo i hms like andom o es s and deep lea ning can be used since hey do no make
assump ions abou he unde lying da a dis ibu ion.
The need o comply wi h binding egula o y equi emen s aimed a p o ec ing ma ine
en i onmen s (a bo h egional and global scale) is ano he mo i a ion o da a science adop ion
in ma ine sciences (Guidi e al., 2020). Examples include: The Ma ine S a egy F amewo k
Di ec i e ha equi es EU membe s a es o achie e Good En i onmen al S a us (GES) in hei
ju isdic ional ma ine en i onmen s e.g by incen i izing ou ine su eys o ma ine ecosys ems o
moni o biodi e si y (Eu opean Commission. Join Resea ch Cen e and In e na ional Council
o he Explo a ion o he Sea (ICES), 2010). Addi ionally, he Eu opean G een Deal ini ia i e
ha aims o ensu e clima e neu ali y in Eu ope by 2050 also equi es manda o y pe iodic
epo ing o he impac s o pollu ion and clima e change on ma ine biodi e si y (Fe ing, n.d.).
Beyond he policy le el, he Eu opean Ma ine Obse a ion and Da a Ne wo k (EMODne ) ha
cu a es au ho i a i e ma ine da ase s also mo i a es he need o enhanced da a p ocessing
capabili ies o da a science algo i hms (Ma ín Míguez e al., 2019). These da a science me hods
would enable EMODne (s akeholde s) o ecien ly ans o m aw ma ine da a in o eliable
ac ionable insigh s ha suppo e idence-based decision making. Ou side o he EU, he
In e na ional Seabed Au ho i y (ISA) ha is esponsible o egula ing deep sea mining o
polyme allic Mn-nodules also has a du y o ensu e ha mining ac i i ies a e conduc ed in an
en i onmen ally sus ainable manne (B äge e al., 2020). In his ega d, he ISA will ine i ably
mo i a e he adop ion o da a science echnologies in ma ine sciences e.g by unding and
acili a ing independen scien ic in es iga ions by academia, equi ing comp ehensi e impac
assessmen epo s om mining companies, and de ec ing unau ho ized mining ac i i ies and
20
po en ial iola ions o he ( o hcoming) Mining Code (Lodge, 2011). These ini ia i es om
he b oade ma ine science communi y will undoub edly mo i a e adop ion o da a sciences.
1.1.2 Da a Science Pe spec i e
Supe ised machine lea ning models a e gene ally ma u e, well-es ablished, and p oduce high
accu acy whene e enough labeled da a is a ailable (Bu ka and Hube , 2021). As a esul ,
hese models eadily nd applicabili y in eg ession o classica ion analysis o (pa ially)
anno a ed ma ine da ase s. In he con ex o ma ine sciences, he anno a ions can be ob ained
ei he in eal ime du ing image acquisi ion (e.g using he OFOP so wa e), o du ing dedica ed
anno a ion sessions h ough pla o ms such as BIIGLE (Langenkämpe e al., 2017). The high
accu acy capabili ies o supe ised models is desi able because i ensu es he eliabili y o
undamen al componen s o ma ine science esea ch e.g seaoo classica ion (Mbani e al.,
2023b) and biodi e si y assessmen (Mbani e al., 2023a). Accu a e quan ica ion o ma ine
biodi e si y and habi a he e ogenei y is essen ial because hese indices suppo e idence-based
decision making in ma ine sciences ega ding e.g es ima ion o spa ial co e age and abundance
o polyme allic Mn-nodules (Schoening e al., 2017), as well as p o ec ion o ulne able ma ine
ecosys ems om an h opogenic ac i i ies such as un egula ed exploi a ion o Mn-nodules
(Glasby, 2002). In addi ion, accu a e baseline da a on habi a s and mega aunal abundance
p o ides a s ong ounda ion upon which ma ine esea che s can ack changes o jus i y ollow
up in es iga ions. Also, supe ised machine lea ning echniques such as Gene alized Linea
Models (GLMs) and Bayesian models allow ma ine esea che s o explici ly speci y he s a is ical
dis ibu ion o he da a o be analyzed (e.g image-de i ed anno a ions) ( an de Schoo e al.,
2021). This abili y o decide he da a dis ibu ion is use ul o modeling in ma ine sciences,
whe e measu ed bio ic and abio ic a iables (mega aunal axa coun s, abundances, empe a u e,
dep h e c) do no always ollow a gaussian dis ibu ion as assumed by mos classical models
(Legend e and Fo in, 1989). Ano he mo i a ion is ha adi ional supe ised machine
lea ning models such as andom o es s and suppo ec o machines a e compu a ionally cheap
(compa ed o e.g deep lea ning) (Li e al., 2016), and also do no equi e a lo o labeled
examples. Conside ing also ha mos o hese adi ional machine lea ning algo i hms a e
designed o ope a e on da a ma ices ( ows and columns), he models nd eady applicabili y in
ma ine da a analysis whe e a ailable anno a ions a e ypically o ganized (o con e ed) in
abula o ma . Finally, aining supe ised models in ol es lea ning a unc ion ha maps om
inpu s o class labels. This aining p ocess is achie ed by op imizing an objec i e unc ion,
which o malizes ou (usually subjec i e) belie s ega ding he na u e o he ela ionship
be ween inpu s and co esponding labels. The e o e, he ac ha i is possible o
(ma hema ically) cus omize his objec i e unc ion o cap u e nuances and assump ions in he
21
ma ine domain is a s ong mo i a ion o inco po a ing supe ised machine lea ning in ma ine
sciences.
Unsupe ised machine lea ning, on he o he hand, is use ul o disco e ing unde lying
s uc u e and hidden pa e ns om unlabeled da ase (Hachaj and Mazu ek, 2020). This is
al eady a s ong mo i a ion o inco po a e unsupe ised me hods in o ma ine science
wo kows, whe e he majo i y o (image and ideo) da ase s a e unlabeled. The abili y o
unsupe ised clus e ing algo i hms o au oma ically g oup oge he isually simila images also
nds eady applicabili y in ma ine sciences. This is because clus e ing educes he complexi y o
ma ine da ase s, which means ha domain scien is s only need o in e p e and assign seman ic
labels o he ew gene a ed clus e s a he han manually inspec ing he en i e da ase . In
ma ine image analysis o example, hese clus e s would ypically be ob ained by s encoding
isual in o ma ion om images on o a po en ial high dimensional ea u e ec o (Bengio e al.,
2014), and hen using e.g euclidean o cosine me ic o measu e how simila he ec o
ep esen a ions a e o each o he in ea u e space. The assump ion he e is ha simila ec o s
(dependen on he chosen me ic) should ideally map close o each o he in ea u e space, and
can he e o e be easily de ec ed as dis inc g oupings using a clus e ing algo i hm like K-means.
This capabili y o clus e images based on ec o simila i y makes unsupe ised me hods eadily
applicable o ma ine sciences, because i allows he he same da ase o be ep esen ed (and
isualized) in mul iple ways depending on he use case e.g based on bio ic composi ion,
subs a e cha ac e is ics, wa e mass p ope ies e ce e a. The only equi emen is ha he
ea u es ex ac ed om images be ep esen a i e and capable o encoding he phenomena o
in e es . Fo example, hand-enginee ed ex u e, colo , and en opy ea u es may be sui able o
ep esen ing he e ogenous seabed co e ed wi h densely dis ibu ed Mn-nodules (Mbani e al.,
2022b), whe eas abs ac ea u es ex ac ed om p e- ained deep lea ning models may be mo e
sui able o cap u ing sub le a iabili y in an o he wise homogeneous seabed o he abyssal
plains (Bengio e al., 2014). The e o e, combining ec o space ep esen a ion wi h clus e ing
echniques allows o quick, bu objec i e so ing o po en ially uns uc u ed ma ine da ase
in o a bi a y ca ego ies. Fu he mo e, he abili y o de ec pa e ns based on ec o
ep esen a ion nds di ec applicabili y in he de ec ion o anomalous pa e ns in ma ine
sciences (e.g a e axa o ne-scale mic ohabi a s), since hese unusual a ibu es would be
g ouped oge he in o a clus e ha de ia es signican ly om he o e all pa e n (Mbani e al.,
2023a). Finally, unsupe ised dimensionali y educ ion echniques such as p incipal
componen s analysis a e able o ecien ly p ojec he high-dimensional ec o ep esen a ions
on o a 2D ea u e space (Mika e al., n.d.). This unlocks he capabili y o g aphically isualize e.g
he en i e seaoo a -a-glance, which is e y appealing in ma ine science domains because i can
in o m decision making e.g whe e o sample nex du ing a c uise, o which egions o he
wo king a ea ha e been o e - o unde -sampled.
22
Weakly supe ised machine lea ning me hods comp ise a ela i ely ecen amily o models wi h
a p omising abili y o lea n pa e ns om noisy o imp ecisely labeled da ase s (Yi e al., 2022).
This abili y nds di ec applicabili y in ma ine science use cases, whe e anno a ions a e a ailable
om p e ious un ela ed s udies, excep no in a o ma ha is machine lea ning- eady. An
example would be mega aunal axa ha we e comp ehensi ely anno a ed on a pho omosaic
using mouse clicks (poin anno a ions), ye aining a s anda d objec de ec ion model equi es
anno a ed bounding box coo dina es. The p omising capabili y o mode n weakly supe ised
models o lea n a signal om such less- han-pe ec anno a ions p o ides a s ong mo i a ion
o conside ing hei inco po a ion in o ma ine science wo kows. This is because he weakly
supe ised models will unlock he po en ial o p oduc i ely e-use ma ine da ase s ha we e
acqui ed, anno a ed, and a chi ed in ins i u ional da a managemen po als o e se e al decades
o sea going ac i i ies.
The wide a ailabili y o well documen ed open sou ce da a analysis amewo ks ha e
signican ly lowe ed he ba ie o implemen a ion o machine lea ning models (Ped egosa e
al., 2011). Tasks such as da a cleaning and explo a o y da a analysis ha p e iously equi ed
w i ing se e al lines o code can now be achie ed h ough s aigh o wa d unc ion calls o
s able and p oduc ion- eady lib a ies such as pandas and ma plo lib (Mckinney, 2011).
Addi ionally, nume ical compu ing lib a ies such as SciPy, sciki -lea n and NumPy expose
ul a-op imised ou ines o da a ma ix manipula ion and s a is ical hypo hesis es ing
(Ped egosa e al., 2011). Recen ly, deep lea ning lib a ies such as PyTo ch and Tenso Flow ha e
abs ac ed away mos o he low-le el coding equi emen s ha p e iously discou aged
adop ion by scien is s om ou side he compu e science domains (Paszke e al., 2019) (Pang e
al., 2020). In pa allel, geospa ial analy ics has also wi nessed ema kable p og ess in he open
sou ce space e.g h ough egula ly upda ed desk op GIS so wa es like QGIS, as well as h ough
well main ained p og amma ic compu ing lib a ies like PyGMT and GeoPandas o ec o
p ocessing, and Ras e io o eading and w i ing geog aphic as e les di ec ly in o NumPy
a ays. Once as e s a e loaded in NumPy o ma , image p ocessing lib a ies such as OpenCV
and o ch ision and sciki -image g ea ly simpli y asks such as supe pixel segmen a ion and
ne- uning o con olu ion neu al ne wo ks. Fu he mo e, amewo ks such as Dask and joblib
ha e simplied he p ocess o scaling pa allelizable wo kows o mul iple CPU co es o e en o
dis ibu ed compu ing clus e s. The in e ope abili y o hese (and o he ) scien ic compu ing
lib a ies (e.g h ough in e media e da a s uc u es like NumPy) ha e also s anda dized da a
science wo kows, which is a s ong mo i a ion o in eg a ion wi h ma ine science wo kows.
23
1.2 S a e o esea ch
1.2.1 Niche o he hesis
This s udy belongs o he wide discipline o emo e sensing o ma ine en i onmen s. In his
con ex , emo e sensing gene ally e e s o he use o a ious imaging (o senso ) modali ies such
as op ical images, ideos and acous ics o documen he ecology, biology and geology o ma ine
ecosys ems. Specically, his hesis concep ualizes and implemen s au oma ed da a science
wo kows ha ecien ly gene a e seman ic anno a ions om la ge sequences o high
esolu ion op ical images, wi h specic case s udies om he Pacic and opical No h
A lan ic. The da a science wo kows implemen ed in his hesis a e also seamlessly in eg a ed
wi h mul i a ia e spa io-ecological echniques in o de o p o ide geog aphic con ex o he
image-de i ed anno a ions. In his way, his hesis p o ides a holis ic amewo k o
cha ac e izing habi a s, mega aunal abundance, as well as en i onmen al d i e s ha inuence
hese pa e ns in ma ine ecosys ems. The e o e, he niche o his hesis is a he in e sec ion o
ma ine sciences and da a sciences (specically compu e ision).
1.2.2 Con ibu ion o science
This hesis makes he ollowing con ibu ions o he ma ine imaging communi y:
a) Image p ocessing: A no el wo kow was de eloped o au oma ically de ec ing lase
poin s om a la ge sequence o deep sea images. The no el y o his implemen a ion lies
in he seamless in eg a ion o image p ocessing (band a i hme ics) and geome ic se
heo y (in e sec ion) me hodologies in o an end- o-end wo kow ha is
compu a ionally as , accu a e and easily scalable h ough concu en execu ion ac oss
mul iple CPU co es. The u ili y o he implemen ed lase poin de ec ion wo kow is
demons a ed he e h ough i s abili y o guide he au oma ic selec ion o a e e ence
image wi h he highes scale o use in colo no maliza ion h ough his og am ma ching
(in he absence o logged came a al i ude in o ma ion). The de ec ed lase poin s also
demons a e applicabili y in he de e mina ion o an op imal e e ence scale ela i e o
which he al i ude-dependen isual oo p in s o espec i e images can be s anda dized.
24
This in ol ed he use o cho ople h maps o quali a i ely pick ou ob ious pa e ns o
geog aphic clus e ing, as well as he use o spa ial au oco ela ion analysis o quan i a i ely map
ou s a is ically signican ho spo s o mega aunal abundance. En i onmen al d i e s ha
inuenced hese obse ed dis ibu ion pa e ns we e also in es iga ed using o dina ion
echniques. The o dina ion in ol ed he use o non-me ic mul idimensional scaling o p ojec
bo h bio ic and abio ic a iables on o a wo dimensional ea u e space, allowing o a
s aigh o wa d g aphical assessmen o in e - ela ionships among he a iables.
Finally, image da ase s, anno a ions, and so wa e code ha we e ei he acqui ed o gene a ed
o m in his hesis ha e all been published o pangaea, gi lab, o as supplemen a y ma e ials
associa ed wi h espec i e pee e iewed publica ions. These esea ch p oduc s a e eely
a ailable o use and e-use by he ma ine scien ic communi y, wi h he aim o p omo ing
anspa ency, open science and collabo a ion o ad ance he discipline o ma ine da a science.
1.4 Ou line o scien ic chap e s and decla a ion o con ibu ion
1.4.1 Scien ic Pape 1: Implemen a ion o an au oma ed wo kow o
image-based seaoo classica ion wi h examples om manganese-nodule
co e ed seabed a eas in he Cen al Pacic Ocean (Mbani e al. 2022)
This chap e p esen s he Au oma ed and In eg a ed Seaoo Classica ion Wo kow
(AI-SCW) ha was implemen ed o pa i ion deep seabed a eas in he Pacic in o seman ic
habi a classes. The classica ion was based on isual in o ma ion ex ac ed om a sequence o
op ical images collec ed along linea ansec s, using he OFOS imaging pla o m ha sampled
he seabed a a cons an equency o 0.1 Hz. This chap e p o ides comp ehensi e desc ip ions
o all cons i uen componen s o he AI-SCW wo kow. These include au oma ic lase poin
de ec ion, de e mina ion o al i ude-dependen image scales, isibili y imp o emen
ans o ma ions, and s anda diza ion o isual oo p in s. T aining schedules and pe o mance
e alua ion me ics ( o bo h supe ised and unsupe ised seaoo image classie s) a e also
p o ided. Finally, spa ial dis ibu ion pa e ns o he obse ed seaoo habi a s a e analyzed in
he con ex o mul ibeam ba hyme y (and i s de i a i es).
I concep ualized he wo kow, w o e he so wa e code o all he image p ocessing, seaoo
classica ion, geospa ial analysis asks. I also w o e he manusc ip .
31
1.4.2 Scien ic Pape 2: An au oma ed image-based wo kow o de ec ing
megaben hic auna in op ical images wi h examples om he
Cla ion–Clippe on Zone (Mbani e al. 2023)
This chap e comp ehensi ely desc ibes he Megaben hic Fauna De ec ion wi h Fas e R-CNN
(FaunD-Fas ) wo kow, which was implemen ed o au oma ically de ec , localize and classi y
mega auna om seaoo images o he Pacic. Fi s , an inno a i e me hodology is p esen ed
ha semi-au oma ically gene a es weak mega aunal bounding box anno a ions based on analysis
o anomalous supe pixels. Subsequen sec ions desc ibe he s eps o pos p ocessing he
anomalies o emo e alse posi i es, allowing o manual assignmen o seman ic mo phospecies
labels o only he uly anomalous supe pixels. Addi ional sec ions desc ibe how he au o
gene a ed anno a ions we e used o ain a s a e-o - he-a ben hic auna de ec ion model, as
well as how he pe o mance o he ained model was e alua ed based on he s anda d COCO
de ec ion me ics. The de eloped model was also benchma ked agains compa able
s a e-o - he-a mega auna de ec ion models. Fu he de ails a e p o ided o he con e sion o
absolu e mega auna coun s in o abundances, as well as cha ac e iza ion o he spa ial
dis ibu ion pa e ns o bio a in he con ex o s uc u ing en i onmen al d i e s.
I concep ualized he wo kow and w o e he so wa e code o all he image p ocessing, ben hic
auna de ec ion, and geospa ial analysis asks. I also w o e he manusc ip .
1.4.3 Scien ic Pape 3: Au oma ed image-based wo kows e eal he
composi ion and spa ial dis ibu ion pa e ns o megaben hic communi ies
in he opical No h A lan ic Ocean (Mbani e al., In e iew SciRep).
This chap e in eg a es he seaoo classica ion and mega auna de ec ion wo kows om he
s wo scien ic chap e s. Since bo h wo kows we e o iginally de eloped and es ed agains
seaoo images om he Pacic, one main objec i e o his s udy was o in es iga e he
gene alizabili y o he wo kows when applied o a new da ase om he opical A lan ic.
Ra he han ocusing on echnical algo i hmic de elopmen , his chap e ocuses mo e on
de ailed in es iga ions o spa io-ecological dis ibu ion pa e ns o anno a ed ben hic habi a s
and mega aunal axa. Desc ip ions a e s p o ided o he geologic ea u es, geog aphic
ex en s and wa e mass p ope ies o he new wo king a ea in he A lan ic, ollowed by a b ie
desc ip ion o he aining, e alua ion and in e ence se ups o he wo anno a ion wo kows.
Nex , he magni ude o sampling, scaling and (mega aunal) double coun ing biases along
espec i e came a deploymen acks is in es iga ed. De ails a e also p o ided o he solu ion o
hese biases h ough he es ablishmen o xed-size sampling uni s wi hin o no malize absolu e
32
mega auna coun s ela i e o ac ual obse ed seaoo a ea. A comp ehensi e desc ip ion o he
use o spa ial au oco ela ion analysis o e eal local ho spo s o mega aunal abundances is also
p o ided, along wi h desc ip ions o how o dina ion echniques based on non-me ic
mul idimensional scaling we e used o g aphically assess he in e - ela ionships be ween bio ic
and abio ic a iables in ea u e space.
I concep ualized he wo kow and w o e he so wa e code o all he image p ocessing, seaoo
classica ion, ben hic auna de ec ion, and geospa ial analysis asks. I also w o e he manusc ip .
33
2. Scien ic Chap e s
2.1 Implemen a ion o an au oma ed wo kow o image-based seaoo
classica ion wi h examples om manganese-nodule co e ed seabed a eas in
he Cen al Pacic Ocean
Me ada a: Mbani, B., Schoening, T., Gazis, IZ. e al. Implemen a ion o an au oma ed
wo kow o image-based seaoo classica ion wi h examples om manganese-nodule co e ed
seabed a eas in he Cen al Pacic Ocean. Sci Rep 12, 15338 (2022).
h ps://doi.o g/10.1038/s41598-022-19070-2
Mo i a ion and Objec i es: Classica ion o he seaoo in o habi a ca ego ies is c ucial
owa ds he es ablishmen o comp ehensi e ma ine baseline in o ma ion. Accu a e and
egula ly upda ed baseline da a in u n suppo s in o med decision making in use cases such as
delinea ion o ma ine p o ec ed a eas, assessmen o en i onmen al impac s o deep sea mining,
as well as acking changes in ma ine ecosys em heal h. Whe eas la ge scale seaoo mapping is
ypically pe o med using medium esolu ion acous ic image y e.g om mul ibeam
echosounde s, op ical imaging is he p e e ed me hod o de ailed close- ange in es iga ion o
selec ed si es o in e es . In his ega d, op ical images can se e as g ound- u hs o e alua ing
he accu acy and eliabili y o acous ics-based mapping me hods, while also being used
independen ly o e eal sub le a iabili y in habi a cha ac e is ics and o he localized
sedimen ological pa e ns (e.g along su ey ansec s) ha may no be de ec able a he
esolu ion o acous ics image y.
Howe e , i is cos ly and in easible o manually anno a e he huge olumes o op ical seaoo
images ha a e nowadays acqui ed du ing scien ic expedi ions. The e o e, one p ima y
objec i e o his s udy was o de elop an au oma ed image-based wo kow o ecien ly
anno a ing seaoo images in o de o e eal sedimen pa e ns in he Mn-nodule co e ed deep
seabed a eas o he Cla ion Clippe on Zone. Specic objec i es included:
1. To au oma e he de ec ion o lase poin s om e e y image. This was use ul o
de e mining al i ude-dependen image scales o be used o con e ing measu emen s
(e.g isual oo p in s) om pixels o eal wo ld me ic uni s (e.g squa e me e s). The
dis ibu ion o image scales was also used o de e mine he e e ence median scaling
ac o ela i e o which all images would be esized, ensu ing a consis en isual
oo p in ep esen a ion on he seabed.
34
2. To colo no malize and imp o e he o e all isual quali y o aw images. The ocus he e
was o accoun o deg ada ions esul ing om wa eleng h-dependen a enua ion and
sca e ing eec s o he p opaga ing LED ligh ( om he imaging pla o m) h ough he
wa e column, as well as inconsis encies in scene b igh ness caused by he a ying
al i ude o he imaging pla o m du ing image acquisi ion.
3. To (semi) au oma e seaoo classica ion by mapping each image in o one o p edened
seaoo habi a classes. This in ol ed deploying bo h supe ised and unsupe ised
image analysis wo kows ha we e each ained based on aining examples ob ained
semi-au oma ically e.g h ough nea es neighbo sampling in ea u e space ( o
supe ised classica ion), as well as s a ied clus e -based sampling ( o unsupe ised
classica ion).
4. To cha ac e ize spa ial dis ibu ion pa e ns o he obse ed seaoo sedimen pa e ns.
This in ol ed in eg a ing he anno a ions wi h o he auxilia y geo e e enced da ase s
(e.g dep h, slope and e ain uggedness), wi h he aim o p o iding geog aphic con ex
o image-de i ed anno a ions e.g by isualizing clus e ing pa e ns on cho ople h maps
plo ed along came a deploymen acks, as well as h ough s a is ical assessmen s o he
inuence o en i onmen al d i e s on he obse ed pa e ns.
Ma e ials and Me hods:
This s udy used 40,678 high esolu ion seaoo images om he Cla ion-Clippe on Zone
(CCZ) in he Pacic. The images we e acqui ed using a Canon EOS 5D Ma k IV came a
a ached o he Ocean Floo Obse a ion Sys em (OFOS) du ing SONNE c uise SO268 o he
Ge man and Belgian con ac a eas o Mn-nodule explo a ion. Twel e ideo in es iga ions
conduc ed in he wo a eas co e ed a combined ack leng h o 92.5 km a an a e age wa e
dep h o 4,280 me e s. Wi hin he Ge man a ea, a chain d edge was used o dis u b he
sedimen as pa o a small-scale expe imen o simula e he po en ial spa io- empo al ex en o
e-deposi ioned sedimen plume. The seaoo in he Ge man con ac a ea was he e o e
pho og aphed wice (be o e and a e he dis u bance expe imen ).
Au oma ic lase poin de ec ion in ol ed a combina ion o image p ocessing and geome ical
analysis. Fi s , h ee isible lase poin s we e anno a ed om a well illumina ed e e ence image.
A 250-pixel bue was hen es ablished a ound he iangula geome y dened by he h ee
35
lase poin s, wi hin which lase poin s om all he images we e expec ed o be loca ed. Nex , an
image p ocessing pipeline was implemen ed h ough a linea combina ion o (RGB) channels,
which p oduced an in e media e signal (image) o which a local-peak nding algo i hm
iden ied po en ial candida e lase poin loca ions. Finally, he ac ual lase poin s we e ob ained
by in e sec ing he bue mask and he candida e lase poin coo dina es. The image scale was
de e mined as he a io be ween he a e age dis ance sepa a ing he de ec ed lase poin s (in
pixels) o hei calib a ed dis ance (o 40 cen ime e s).
Illumina ion and colo no maliza ion in ol ed ou key s eps. Fi s , he ligh cone eec was
co ec ed by pixelwise z-sco e no maliza ion applied o ba ches o sequen ially o de ed images.
Second, image con as was maximized using adap i e his og am equaliza ion ha edis ibu ed
image in ensi ies in local image iles esul ing in imp o ed o e all image con as . Thi d, une en
b igh ness among images was co ec ed by adjus ing he in ensi y dis ibu ion (his og ams) o
all images o ma ch he his og am o a manually chosen e e ence image (wi h good o e all
scene b igh ness). Finally, he images we e escaled ( ela i e o he median scale) and hen cen e
c opped o ep esen s anda dized isual oo p in s (o 1.6 squa e me e s each) on he seabed.
Supe ised seaoo classica ion in ol ed ne- uning an ins ance o he Incep ion V3
con olu ional neu al ne wo k, and hen applying he ained model (in in e ence mode) o
p edic habi a class labels o espec i e images. The habi a classes included: Class Seafloo A
ha comp ised images o which he seabed was co e ed wi h no o only ew Mn-nodules.
Images om his class also exhibi ed isible d edge ma ks and u ned-o e sedimen indica i e
o he impac om he chain d edge expe imen ; Class Seafloo B comp ised pa chy
Mn-nodules ha we e quali a i ely small sized and only pa ially co e ed he seabed; Class
Seafloo C comp ised Mn-nodules ha we e densely dis ibu ed pe uni a ea; and class Seafloo
Dcomp ised quali a i ely la ge sized Mn-nodules ela i e o hose in classes Seafloo B and C.
On he o he hand, unsupe ised classica ion was achie ed by applying K-means clus e ing o
he da a ma ix o six-dimensional ex u e and en opy ea u es ex ac ed om he en i e
images. The clus e ing g ouped oge he isually simila images, a e which a domain expe
manually inspec ed he clus e ing (and co esponding image subsamples) in o de o assign each
clus e a seman ic class label.
Pe o mance o he classica ion models was e alua ed using s anda d p ecision, ecall and
con usion ma ix ( o he supe ised Incep ion V3), while he silhoue e sco e was used o assess
he quali y o clus e ing ( o he unsupe ised K-means).
Cho ople h maps (based on pho o cen e coo dina es) we e used o isually assess he spa ial
dis ibu ion pa e ns o seaoo habi a s along came a deploymen acks. In addi ion, boxplo s
36
we e also gene a ed o show he a iabili y o en i onmen al d i e s in espec i e di es, as well
as he co ela ion be ween he d i e s and die en habi a classes.
Key Findings:
The colo no maliza ion wo kow imp o ed he isual quali y o images by (a) signican ly
educing he isual eec o g adual educ ion o ligh owa ds he image edges, (b) maximizing
he dis ibu ion o pixel in ensi ies o span he en i e 8-bi dynamic ange, and (c) elimina ing
he g eenish haze o imp o e he o e all sha pness and cla i y o images.
Howe e , i was obse ed ha lase poin s we e de ec ed wi h high accu acy om aw images
compa ed o he colo no malized images ha p oduced many alse posi i es. This obse a ion
was he esul o a side eec o he applied adap i e his og am equaliza ion ans o ma ion,
which desi ably imp o ed he o e all image con as bu in he p ocess educed he
dis inc i eness o he ed lase poin s ela i e o local backg ound pixels. Ano he explana ion
was ha he colo no maliza ion ans o ma ion could ha e in oduced (o amplied) noise in
espec i e images, which educed he signal- o-noise a io o he ed lase poin s, he eby
making he lase s indis inguishable om he backg ound.
Figu e 9: Example images showing isibili y imp o emen ans o ma ion. (A) shows a well
illumina ed e e ence image whose (B) in ensi y dis ibu ion was used o ans o m (C) a aw
37
image in o (D) ans o med image
Addi ionally, he use o he same xed-size iangula bue mask o l e down o he h ee
ac ual lase poin s om a pool o candida e lase loca ions allowed o a s aigh o wa d
concu en implemen a ion o he lase de ec ion wo kow. This is because lase poin
de ec ions in espec i e images occu ed independen o each o he , which made i possible o
compu a ionally map he wo kow o mul iple CPU co es simul aneously. I was obse ed ha
his app oach inc eased p ocessing speeds (by up o a ac o o 3) compa ed o al e na i e
me hods in ol ing sequen ial p ocessing.
P ojec ing seaoo images o ea u e space and hen pe o ming nea es neighbo sampling
a ound a ew manual anno a ions signican ly expedi ed he p ocess o gene a ing labeled
aining examples o supe ised classica ion. This app oach also allowed o a quick
a -a-glance isualiza ion o seabed cha ac e is ics, which enabled expe anno a o s o ac o in
ich and nuanced con ex ual in o ma ion abou e.g he p ope ies o neighbo ing clus e s, o
whe he in e -class ansi ion pa e ns a e sub le o ab up . Such unde lying con ex ual
seaoo pa e ns and ela ionships may be missed when images a e manually inspec ed
sequen ially (in isola ion). In addi ion, he ac ha he p ojec ion o ea u e space is
de e minis ic implies ha he anno a ion wo kow can easily be gene alized o o he seabed
a eas, p o ided ha he isual ea u es ex ac ed om images accu a ely cap u e he sedimen
pa e ns. Fo example, domain knowledge on he Mn-nodule a iabili y and appea ance (as
app oxima ely ound blobs o da k pixels) may lead o he choice o ex u e and en opy as
app op ia e ea u es o encode isual in o ma ion om seaoo images o he CCZ. This
p ojec ion-based app oach also p o ides a na u al way o p e en class imbalance, since
anno a o s can easily ( isualize and) o e sample egions o he seabed (o ea u e space) ha a e
unde ep esen ed, while uni o mly sampling he es o he ea u e space o ensu e balanced
ep esen a ion ac oss habi a classes. While image ea u es ex ac ed using p e- ained models
a e in gene al e y exp essi e, om his hesis indica e ha in si ua ions whe e one has an idea
o he seaoo cha ac e is ics (ei he a e isualizing p ojec ions in ea u e space p ojec ion, o
om p io domain knowledge), hen manual ea u e selec ion may p oduce supe io
classica ion esul s especially in unsupe ised se ing.
38
Figu e 10: P ojec ion o he seaoo in o ea u e space, colo coded by espec i e seaoo
classes. No ice how he unsupe ised classica ion p oduces ab up class bounda ies.
Randomly sampling images om a ba ch o seaoo images o ain an unsupe ised K-means
classie is he bes s a egy only i he me ic being op imized agains is compu a ional speed.
The downside o andom sampling is he class imbalance ha may a ise when dominan egions
o he seabed a e o e ep esen ed in he sample, conside ing ha subs a e pa e ns in he deep
sea a y ela i ely slowly (in kilome e scale). T aining models wi h a class-imbalanced aining
se esul s in poo gene aliza ion pe o mance, o misleading conclusions. S a ied
clus e -based sampling was ound o be he bes s a egy when he me ic o compa ison was he
quali y o clus e g oupings (as measu ed by he silhoue e sco e). This was because he ini ial
o e clus e ing s ep allows o a mo e uni o m and ep esen a i e sampling o he seabed,
including anomalous egions e.g egions o he Cla ion Clippe on Zone whe e he e a e no
deposi s o polyme allic Mn-nodules. Rega dless o he chosen sampling s a egy, ndings
indica e ha unsupe ised classica ion gene ally p oduces ab up class bounda ies (in ea u e
space), which is no ypical o deep sea en i onmen s whe e in e -class ansi ions a e smoo h
and g adual. Close in es iga ions e ealed ha hese sha p class bounda ies we e caused by he
39
K-means objec i e unc ion, whose op imiza ion minimizes he wi hin-clus e sum o squa es
esul ing in dis inc and well-dened clus e s. In con as , he non con ex objec i e unc ions
ha a e used in deep lea ning models (e.g c oss-en opy loss) na u ally p oduce uzzy class
bounda ies. Mo eo e , deep lea ning models ypically ou pu p obabili y dis ibu ions o e
p edic ed class labels a he han single de e minis ic class labels; hese dis ibu ions can also be
used o uzzi y class bounda ies. S ill, bo h esul s o supe ised and unsupe ised seaoo
classica ions exhibi ed good ag eemen o e all, as e idenced by a Cohen's Kappa coecien o
0.6 ha indica es he classica ions a e mo e consis en han would be expec ed om chance.
40
he pa ame e inuences how sensi i e he model is o anomalies. Thus, al hough se ing he
con amina ion ac o o a high alue p oduced many alse posi i es, his choice was p e e able in
his s udy since i allowed domain expe s o be di ec ly in ol ed e.g o closely inspec alse
posi i es, o o de e mine whe he a supe pixel showing wha appea s o be a pa ially
bu owed ophiu oid is indeed anomalous.
The au o gene a ed weak bounding box anno a ions we e no sucien (in quali y and
quan i y) o a comp ehensi e cha ac e isa ion o ben hic biodi e si y. This is because he weak
anno a ions we e gene a ed using an unsupe ised (supe pixel-based) app oach o which alse
nega i es a e highly p obable.
The e o e, such weak anno a ions should be ega ded as compu a ionally cheap sou ces o
aining examples o be la e ened, seman ically labeled, and e en ually used o ne- uning a
s a e-o - he-a objec de ec ion model. Fo example, his s udy ained a Fas e R-CNN model
using seman ically labeled bounding box anno a ions and achie ed an accu acy o 78.1% (a an
in e sec ion-o e -union h eshold o 0.5), which was on a pa wi h o he s a e-o - he-a
ben hic objec de ec o s ha we e ained using manually anno a ed aining examples. This
good pe o mance demons a es ha e en hough supe pixel-based anno a ions a e
compu a ionally cheap o gene a e, hey a e eliable and he e o e easily scalable o e en la ge
image da ase s as long as he e is sucien compu ing powe . This supe pixel-based app oach o
expedi ed anno a ion eadily nds applicabili y in en i onmen s whe e compu a ional capaci y
may be limi ed e.g on scien ic expedi ions on boa d a small esea ch essel, o in eal- ime
analysis o li e ideo eed. Benchma king wi h o he compa able s a e-o - he-a models showed
ha while wo s age de ec o s like Fas e R-CNN p oduce high accu acy, he models a e slow
(in compa ison) and he e o e no sui able in ce ain applica ions e.g on edge de ices like
NVIDIA je son. Ano he key nding was ha de ec ion o small-sized mega auna was
consis en ly challenging ac oss all he benchma ked ben hic objec de ec o s. This dicul y
could be because small-sized mega auna occupy ewe pixels and a e he e o e easily occluded o
camouaged ela i e o he backg ound en i onmen . Ano he eason could be he echnical
design o he ne wo k, in which he sequence o con olu ion and pooling ope a ions g adually
educe he esolu ion o he ea u e maps. This downsampling ul ima ely leads o he
disappea ance o he small-sized mega auna deepe in o he ne wo k.
47
Figu e 14: Examples o anno a ed mega auna ha we e au oma ically de ec ed using he
ained Fas e R-CNN model. Also shown (in ed) is he se o alse nega i es comp ising
mos ly small-sized objec s and hose ha a e isually simila o backg ound seaoo .
Mega aunal abundance was gene ally low (< 1 ind. pe sq. m) in he su eyed a ea o he Cla ion
Clippe on Zone. In pa icula , egional compa ison showed ha he Ge man con ac a ea
exhibi ed highe mega aunal abundances (0.247 ind. pe sq. m) when compa ed o he Belgian
con ac a ea (0.200 ind. pe sq. m). The Ge man a ea also exhibi ed highe di e si y o
mega auna, wi h a Shannon di e si y index o 2.4 compa ed o he Belgian a ea (1.7).
Conside ing ha he Ge man a ea is on a e age shallowe (-4121 m) compa ed o he Belgian
a ea (-4510 m), his a iabili y in bo h abundance and di e si y could be explainable by he
Eas - o-Wes educ ion in POC ux ha a ails mo e ood o he Ge man seabed in he o m o
sinking o ganic ma e ial.
48
Figu e 15: Map showing he spa ial dis ibu ion o mega aunal abundances along espec i e
su ey ansec s in he Cla ion-Clippe on Zone o he Pacic.
Fu he in es iga ions e ealed ha he majo i y (68%) o mega auna in he Belgian a ea
occupied egions o he seaoo co e ed wi h la ge sized Mn-nodule seabed, despi e he egion
being p edominan ly co e ed wi h bo h la ge sized and densely dis ibu ed Mn-nodules pe
uni a ea. The hypo hesis he e is ha densely dis ibu ed Mn-nodules co e mos o he seabed
49
su ace a ea, which does no allow sucien space o so sedimen dwelle s. This is in
compa ison o la ge sized Mn-nodules ha lea e mo e space among hemsel es, he eby c ea ing
ne-scale he e ogenei y ha accommoda es bo h epi auna and so -sedimen dwelle s such as
c us aceans and echinode ms. Ano he nding was ha ophiu oids and xenophyopho es
exhibi ed high abundance and di e si y in bo h he Ge man and Belgian con ac a eas. The
co-occu ence o hese mo phospecies could be because xenophyopho es c ea e complex
s uc u es on he seabed ha p o ides habi a and shel e o he ophiu oids. The ophiu oids
may also be a ac ed o he o ganic ma e ials accumula ed in xenophyopho e es s.
2.3 Au oma ed image-based wo kows e eal he composi ion and spa ial
dis ibu ion pa e ns o megaben hic communi ies in he opical No h
A lan ic Ocean.
Me ada a: Mbani, B., & G eine , J. (In Re iew, SciRep) h ps://doi.o g/10.31223/X5CQ6B
Mo i a ion and Objec i es: De ailed documen a ion o mega aunal communi ies and hei
habi a cha ac e is ics is key owa ds he o e all moni o ing o ma ine ecosys em unc ioning.
Al hough he e ha e been pas s udies ha success ully used da a science echniques o
subs a e cha ac e isa ion and ma ine biodi e si y assessmen s, mos o hese models ha e no
been ex ensi ely es ed in seabed a eas o he han whe e hey we e o iginally ained. As a esul ,
he obus ness and gene aliza ion capabili ies o hese au oma ed wo kows when applied o
a ying subs a e ypes, wa e masses, and mega aunal communi ies emains unknown. In
addi ion, many s udies ha de elop da a science-based wo kows o ben hic cha ac e isa ion
ypically ocus mo e on echnical aspec s such as ne wo k a chi ec u e design, op imiza ion
s a egies, and pe o mance benchma king o demons a e (some imes ma ginal) imp o emen s
agains exis ing s a e-o - he-a models. Few s udies go u he o p o ide spa io-ecological
in e p e a ions o he image-de i ed anno a ions. As a esul , he e is a need o comp ehensi e
esea ch ha in eg a es bo h aspec s o echnological ad ancemen s as well as spa io-ecological
in e p e a ions.
The e o e, a majo objec i e o his s udy was o in es iga e he gene aliza ion capabili ies o
wo au oma ic image-based wo kows o habi a cha ac e iza ion and mega auna anno a ion.
Bo h wo kows we e o iginally de eloped and es ed agains images o he Pacic (see pape s 1
and 2 abo e). Fo his s udy, howe e , he models we e applied o a new da ase om he
opical No h A lan ic.
50
Thus, specic objec i es o his s udy included:
1. To au oma ically anno a e habi a s and mega aunal axa in deep seabed a eas o he
opical No h A lan ic using ne- uned da a science-based wo kows om p e ious
ela ed s udies (see pape s 1 and 2).
2. To in es iga e po en ial mani es a ions o sampling and scaling biases along came a
deploymen acks, as well as o accoun o hese biases by pooling and no malizing
anno a ions ela i e o xed-leng h sampling uni s.
3. To e eal s a is ically signican ho spo s, coldspo s and o he dis ibu ion pa e ns o
mega aunal abundance based on analysis o spa ial au oco ela ion.
4. To in es iga e he ela i e con ibu ions o he die en axa g oups owa ds simila i y
and/o dissimila i y o mega aunal communi ies.
5. To in es iga e he inuence o en i onmen al d i e s (e.g dep h, slope, empe a u e and
e ain uggedness) on he obse ed mega aunal dis ibu ion pa e ns.
Ma e ials and Me hods:
The da ase o his s udy comp ised 8,838 high esolu ion seaoo images om an Eas -Wes
sec ion o he opical A lan ic loca ed osho e Mau i ania and No h o Cape Ve des. The
images we e collec ed using an XOFOS du ing c uise M182, whose o e all aim was o
in es iga e he inuence o mesoscale eddies on biogeochemical p ocesses and modula ion o
o ganic ca bon anspo in he Eas e n bounda y upwelling sys ems.
P e iously de eloped Au oma ed and In eg a ed Seaoo Classica ion Wo kow (AI-SCW)
was ained using a subse o his new image da ase om he A lan ic, and hen applied in
in e ence mode o p edic habi a class labels o he en i e da ase . Specically, he
unsupe ised classie implemen ed in AI-SCW was used o clus e seaoo images in o na u al
g oupings based on simila i y in isual ea u es, a e which he clus e s we e manually inspec ed
and assigned seman ic labels. In addi ion o he AI-SCW, p e iously de eloped Fauna
51
De ec ion wi h Fas e R-CNN (FaunD-Fas ) wo kow was deployed as-is o au oma ically
de ec mega auna om he sequence o images. Since FaunD-Fas was also o iginally ained
using images om he Pacic, he de ec ed objec s au oma ically became weak anno a ions ha
needed o be manually ened, seman ically labeled, and e en ually used as aining examples
o e aining FaunD-Fas . This e ained e sion o FaunD-Fas is he model ha was used o
de ec , localize, classi y and coun ins ances o mega aunal axa om he en i e image da ase .
G aphs showing a iabili y in numbe o images, sampling speed and obse ed isual oo p in
we e gene a ed o assess po en ial sampling and scaling biases along espec i e su ey ansec s.
In addi ion, he a e age dis ance be ween successi e images was compa ed agains he a e age
leng h o he along- ack image axis o check o po en ial double coun ing o mega auna due o
image o e lap; he e was o e lap i he image leng h was on a e age sho e han he dis ance
sepa a ing he images. To accoun o hese biases, xed-leng h (100-me e -long) sampling uni s
we e dened wi hin which absolu e mega aunal coun s we e con e ed in o abundances ela i e
o ac ual obse ed isual oo p in . Quali a i ely, he spa ial dis ibu ion o mega auna was
isualized on a cho ople h map ob ained by colo -coding cen oid coo dina es o espec i e
sampling uni s based on binned mega aunal abundances. Quan i a i ely, measu es o spa ial
au oco ela ion we e calcula ed o iden i y s a is ically signican ho spo s and coldspo s o
mega aunal abundance. Specically, Mo an’s I sca e plo was used o classi y Local Indica o s
o Spa ial Associa ion (LISA) s a is ics, he eby e ealing egions o he seabed whe e
mega aunal abundances we e signican ly highe o lowe compa ed o a e age local
neighbo hood abundances.
Finally, non-me ic mul idimensional scaling (nm-MDS) o dina ion was used o isually assess
in e - ela ionships among mega aunal axa g oupings and en i onmen al a iables. The
nm-MDS o dina ion in ol ed p ojec ing all he 232 xed-leng h sampling uni s on o a
wo-dimensional ea u e space, and hen supe imposing en i onmen al d i e s (e.g dep h, slope,
opog aphic posi ion index, e ain uggedness, salini y, empe a u e and longi ude) on o he
same ea u e space. The goal o he o dina ion was o iden i y he (subse ) o en i onmen al
ac o s ha po en ially explain he obse ed dis ibu ion pa e ns o mega aunal axa. This
o dina ion was also used o gene a e hypo heses abou he unde lying ecological p ocesses ha
inuence mega aunal communi y s uc u es.
Key Findings:
Au oma ed seaoo classica ion in he opical No h A lan ic e ealed se en clea ly dis inc
habi a classes. On close inspec ion o subsamples, each clus e was ound o con ain images
om he same di e whe e he physical cha ac e is ics o seaoo subs a es was mo e-o -less
simila . Fu he mo e, p ojec ing he images in ea u e space e ealed ha di es ha we e in
close geog aphical p oximi y o each o he also mapped close o each o he in ea u e space.
52
Colo coding he ea u e space p ojec ion wi h clus e ing esul s e ealed a le - o- igh g adien
in he in ensi y o biogenic ac i i y wi hin an o he wise homogeneous seabed. In pa icula , he
in ensi y o ( isible) sedimen ewo king was low o clus e s on he le hal o he ea u e
space, while he deg ee o sedimen dis u bance inc eased signican ly owa ds he igh hal o
he ea u e space. The e o e, he s (ho izon al) axis o he nm-MDS clea ly pa i ioned he
seaoo based on bio u ba ion in ensi y, bu i was no immedia ely ob ious wha a ibu e he
second axis was pa i ioning he seaoo agains . Addi ionally, sub-pa i ions we e also obse ed
wi hin mos o he majo clus e s, which could be in e p e ed as sub le a ia ions in
sedimen ological p ope ies along he came a deploymen acks a sho spa ial scales.
Figu e 16: Fea u e space ep esen a ion o seaoo classes in he opical No h A lan ic.
Images om he Eas e n egion ha showed isible signs o sedimen ewo king due o
bio u ba ion a e g ouped oge he on he igh hal o he ea u e space. Also no e he
subpa i ions wi hin espec i e di es.
A o al o 10,189 o ganisms belonging o 13 axa g oups we e de ec ed ac oss bo h he Eas e n
and Wes e n egions. Al hough ndings showed ha he e was po en ial double coun ing o
53
mega auna in di e 19 due o o e lapping images, his did no aec abundance es ima es
because absolu e axa coun s in espec i e sampling uni s we e no malized by di iding ela i e
o ac ual obse ed isual oo p in s (which would also be doubled because o he o e lap). In
e ms o p opo ions, Fo amini e a, Echinode ma a and Lebensspu en we e he mos dominan
axa ha accoun ed o mo e han 76% o all de ec ions in bo h Eas e n and Wes e n egions.
The es o he axa g oups accoun ed o less han 10% each in ela i e p opo ions, including:
Po i e a, A h opoda, Cnida ia, Sponge-Skele on, Mollusca, Cho da a, Annelida, C enopho a
and Chae ogna ha. Regional compa isons e ealed s a is ically signican die ences in bo h
abundances and di e si y be ween he shallowe close - o-sho e Eas e n egion, and he deepe
Wes e n egion. Specically, he Eas e n egion exhibi ed an a e age abundance o 0.44 (ind. pe
sq. m) compa ed o 0.03 in he Wes e n egion. Va iabili y in mega aunal abundances (also an
indica o o ecological he e ogenei y) was highe in he shallowe Eas e n egion (s anda d
de ia ion = ± 0.35) compa ed o he Wes e n egion (s anda d de ia ion = ± 0.02). Simila i y
pe cen ages (SIMPER) analysis u he e ealed ha 50% o he dissimila i y in bio ic
composi ion be ween he Eas e n and Wes e n egions could be explained by only ou axa
g oups, including: Po i e a (14.46%), Lebensspu en (13.27%), Cnida ia (11.97%) and Mollusca
(9.65%). The high mega aunal abundances and di e si y in he Eas e n egion migh be
explainable by he high ood a ailabili y in he o m o sinking o ganic ma e , as well as by he
ela i ely wa me empe a u es ha enhance me abolic a es o mega auna.
54
Figu e 17: Dis ibu ion o mega aunal abundances o espec i e axa in he opical No h
A lan ic. The shallow Eas e n egion eco ds highe abundances and di e si y among all he
axa g oups.
Mega aunal abundances exhibi ed clea pa e ns o geog aphic clus e ing a bo h local and
egional scale. The Eas e n egion exhibi ed ela i ely high abundances consis en ly h oughou
espec i e obse a ional ansec s (excep in he ela i ely a e ains o di e 131).
Topog aphically complex a eas in he Eas e n egion in pa icula exhibi ed signican ly highe
abundances e.g he sides o he subma ine canyon (in di e 144) as well as he op o he
seamoun in di e 145. The deepe Wes e n egion gene ally exhibi ed low abundances o
mega auna, wi h opog aphically complex ea u es he e simila ly exhibi ing s a is ically
signican ho spo s o mega aunal abundances e.g he op o he seamoun (in di e 32), as well
as he pai o abyssal hills in di e 28.
55
Figu e 18: P ole iew showing s a is ically signican mega aunal ho spo s and coldspo s
along espec i e su ey ansec s in he opical No h A lan ic. Topog aphically complex
egions o he seabed con ained mos mega aunal ho spo s e.g he sides o he subma ine
canyon in di e 144.
P ojec ing he sampling uni s on o a wo-dimensional o dina ion ea u e space e ealed wo
main clus e s o mega aunal communi ies co esponding o he Eas e n and Wes e n egions.
Mega aunal axa ha p ima ily inuenced he deepe Wes e n egion included Echinode ma a,
Fo amini e a, and A h opoda. The emaining en axa p edominan ly inuenced he shallowe
Eas e n egion, po en ially explaining he high numbe o biogenic s uc u es (Lebensspu en)
obse ed in he Eas e n egion. Ano he obse a ion was ha ba hyme ic d i e s such as slope,
uggedness, and oughness p edominan ly inuenced mega aunal communi ies in he deepe
Wes e n egion. This obse a ion could be because opog aphies in he deepe egions a y
slowly in kilome e scale, such ha e en mino a iabili y in ba hyme ic de i a i es wi hin
hese deepe seabed a eas esul s in a mo e p onounced inuence on e.g hyd odynamic eec s
like cu en pa e ns, as well as nu ien dis ibu ion. In con as , he al eady complex
opog aphies in shallowe pa s na u ally dis up hyd odynamic ows, so ha he eec o
mino changes in ba hyme ic de i a i es a e no as p onounced.
56
3. Conclusion, challenges and u u e ou look
This hesis concep ualized, implemen ed and p esen s a sound amewo k o in eg a ing da a
science echniques wi h con en ional ma ine science me hods. Wo kows such as AI-SCW and
FaunD-Fas ha we e de eloped in his hesis ha e so a demons a ed he capabili ies o da a
sciences o au oma e epe i i e and ime-consuming aspec s o image-based ben hic
cha ac e isa ion s udies (e.g h ough accu a e anno a ion o mega auna and sedimen pa e ns).
This au oma ion oe s signican speed gains o ma ine scien is s by educing he human eo
equi ed o execu e ou ine asks, allowing scien is s o ocus on co e scien ic esea ch and
inno a ion. Al hough ully au oma ed end- o-end wo kows a e ecien and e en ideal o
ce ain applica ions e.g pe iodic en i onmen al moni o ing o en o ce egula o y compliance
(o b oadly es ima e p oduc ion capaci y) o deep sea polyme allic Mn-nodule mining
ope a ions (Volkmann and Lehnen, 2018), one key conclusion om his hesis is ha domain
expe ise mus s ill be inco po a ed in o he analy ical wo kows o a numbe o easons: Fi s ,
he complexi y o deep sea en i onmen s in oduces biases ha a y unp edic ably om
expedi ion o expedi ion (o e en wi hin a pa icula di e). As a esul , di ec applica ion o
o- he-shel image p ocessing and da a science echniques (wi hou expe inpu om ma ine
scien is s) will ei he in oduce non-ob ious a i ac s, o lead o ou igh ly misleading
conclusions. Fo example, he high deg ee o sedimen dis u bance due o biogenic ac i i y in
he opical No h A lan ic may al e he composi ion o suspended pa icles and o ganic
ma e in he wa e column, which in u n inuences he op ical p ope ies o he wa e (and
he e o e image quali y) die en ly compa ed o he ela i ely s able and less dis u bed sedimen
in he Pacic; Second, an objec i e de e mina ion o he quali y and comple eness o
au ogene a ed ben hic anno a ions equi es ounda ional unde s anding o geologic con ex , as
well as spa io-ecological p ocesses ha inuence in e ac ions be ween bio ic and abio ic
a iables. Fo example, while he use o anomalous supe pixels as weak bounding box
anno a ions did expedi e he o e all p ocess o gene a ing mega aunal anno a ions, ailu e o
inco po a e domain expe ise may easily ha e caused ce ain mo phospecies o be misclassied
as alse nega i es e.g ophiu oids ha pa ially bu ow in sedimen o eed on o ganic de i us
(Wa ne , 1982) un il hey isually almos esembled backg ound seaoo ; Thi d, he se ious
nancial and legal consequences ha accompany non-compliance wi h egula o y equi emen s
(e.g E.U Habi a s Di ec i e (E ans, 2006)) does necessi a e he inco po a ion o ma ine domain
expe ise o p o ide independen o e sigh o au oma ed ben hic cha ac e isa ion wo kows.
Collec i ely, his hesis shows ha in eg a ion o Ma ine and Da a Sciences should be
complemen a y in na u e. Specically, da a science echniques should be deployed o ecien ly
ans o m po en ially uns uc u ed aw ma ine big da ase s in o in o ma ion, whe eas he ask
o p o iding ich, nuanced and con ex ual in e p e a ions o he au o gene a ed in o ma ion
mus be ese ed o ma ine domain expe s.
63
Howe e , he adop ion o da a science echnologies in con en ional image-based ma ine science
wo kows is s ill ela i ely low. This is due o a numbe o open challenges: Fi s , aw
unde wa e images a e ypically no di ec ly usable as inpu s o da a science models because he
images sue om deg ada ions and o he biases e.g due o a iabili y in sampling gea s,
acquisi ion al i ude, ligh ing condi ions, wa e mass p ope ies e ce e a. The choice be ween
whe he o accoun o hese deg ada ions sepa a ely, o o di ec ly de elop gene alizable
wo kows ha a e by-design agnos ic o he abo e a iabili ies in oduces ex a laye s o
complexi y ha slows down adop ion. Fu u e wo k could explo e po en ial da a-cen ic
app oaches ha o ganically inco po a es ma ine science assump ions and nuances in o he co e
o neu al ne wo k a chi ec u es e.g by s a egically o mula ing objec i e unc ions o be
op imized; Second, na iga ing s ingen egula o y landscape (e.g he EU A icial In elligence
Ac (Veale and Bo gesius, 2021)), as well as compe ing (in e )na ional in e es s may also slow
down he mains eaming o da a sciences echnologies in o ma ine sciences, especially in
sensi i e (o con es ed) applica ions in ol ing e.g g an ing o ejec ion o Mn-nodule mining
licenses in he Cla ion-Clippe on Zone. Fu u e wo k could explo e he easibili y o exible
sandbox app oaches ha allow o con olled es ing o he s eng hs, weaknesses and isks o
eme ging da a science echnologies ela i e o ag eed-upon pa ame e s (e.g da a p i acy), wi h
he expe imen a ion being conduc ed unde a elaxed e sion o supe ision by egula o s
(T uby e al., 2022). Lessons lea ned om hese con olled expe imen s could hen in o m
u u e egula o y decisions; Thi d, he lack o comp ehensi e s anda dized benchma k da ase s
o ad ancing ma ine da a science esea ch also poses adop ion challenges. Specically, i is no
ye ob ious how o accoun o he concep d i (Langenkämpe e al., 2020) ha a ises om
he die en sampling s a egies employed by scien is s who collec , anno a e and e en ually
publish hei labeled da ase s. Fu u e wo k could explo e he de elopmen o open p o ocols
ha s anda dize sampling campaigns, as well as possibili ies o ecien ly gene a ing syn he ic
aining da ase s based on physics-based ende ing (Nimie -Da id e al., 2019) ha ai h ully
ep oduce complexi ies in ma ine en i onmen s; Fou h, in eg a ion o da a science echniques
in o es ablished ma ine science wo kows may no be ob ious in ce ain si ua ions, ei he
because o incompa ible da a o ma s o non-aligned app oaches o anno a ion be ween he
ma ine and da a science communi ies. Fo example, anno a ions o be used o aining da a
science models a e equi ed o be s a is ically balanced ac oss he die en axa classes, which is
no necessa ily a conside a ion in ma ine sciences whe e he ocus is mo e on e ealing exis ing
pa e ns and o mula ing hypo heses abou unde lying p ocesses. Ano he example o
non-alignmen is ha ‘anno a ions’ in da a science ypically means he chosen subse o he
en i e da ase o be used as aining examples, whe eas in ma ine sciences anno a ions a e he
ac ual seman ic in o ma ion ha is de i ed om aw images. In addi ion, he e is usually a
de aul assump ion in da a science wo kows o esize (o c op) images o con o m o he
s anda d dimensions o he inpu laye o he con olu ional neu al ne wo k a chi ec u e. While
64
i ial in da a science wo kows, his a bi a y esizing ope a ion is consequen ial in ma ine
sciences because i in e e es wi h he seman ic in e p e abili y o image con en e.g ac ual
isual oo p in on he seabed. The e o e, u u e wo k could explo e inno a i e s a egies o
es ablishing a common ame o e e ence be ween ma ine and da a science domains e.g by
ha monizing e minologies, assump ions, and da a o ma s in a way ha acili a es seamless
bidi ec ional da a exchange be ween da a science models and spa io-ecological wo kows.
Fu u e ou look sugges s ha ma ine sciences will con inue o expe ience a su ge in ma ine big
da ase s, along wi h a co esponding inc eased demand o ma ine da a analy ical expe ise.
This is e idenced by he e e g owing global consensus on he cen al ole o he oceans in
add essing mode n socie al challenges such as clima e change and g een ene gy ansi ion (Bigg
e al., 2003). Recommenda ions om o wa d-looking policy documen s such as he Eu opean
Ma ine Boa d’s Fu u e Science B ie s (“Fu u e Science B ie s | Eu opean Ma ine Boa d,” n.d.)
s ongly ad oca e o inc eased unding o collabo a i e, ansdisciplina y, and c oss cu ing
pa ne ships o suppo ma ine science esea ch ac i i ies e.g h ough p og ams such as
Ho izon Eu ope (Tenhunen-Lunkka and Honkanen, 2024). In addi ion, he possible adop ion
o he In e na ional Seabed Au ho i y’s Mining Code o egula e deep-sea mining ac i i ies
(Singh, 2021) will undoub edly d i e up demand o ma ine da a science wo kows o suppo
e.g e idence-based moni o ing o egula o y compliance, as well as objec i e en i onmen al
impac assessmen s by s akeholde s (Peuke e al., 2018). In gene al, such policy ini ia i es will
incen i ize u u e in es men s in in eg a ed ma ine big da a handling and analy ical wo kows
cen e ed a ound o e a ching hemes such as: (a) long- e m moni o ing o ma ine ecosys em
unc ions, se ices and biodi e si y; (b) esponsible na u al esou ce exploi a ion e.g
polyme allic Mn-nodules o balance ecological, economic and social in e es s; (c) maximizing
he alue o ede a ed da a po als ha adhe e o Findable, Accessible, In e ope able and
Reusable (FAIR) p inciples (Schoening e al., 2018)
65
Bibliog aphy
Albano, P.G., Azza one, M., Ama i, B., Bogi, C., Sabelli, B., Rilo , G., 2020. Low di e si y o
poo ly explo ed? Mesopho ic molluscs highligh unde sampling in he Eas e n
Medi e anean. Biodi e s. Conse . 29, 4059–4072.
h ps://doi.o g/10.1007/s10531-020-02063-w
Bakke , J.D., 2024. Types o O dina ion Me hods.
Bengio, Y., Cou ille, A., Vincen , P., 2014. Rep esen a ion Lea ning: A Re iew and New
Pe spec i es.
Bigg, G.R., Jickells, T.D., Liss, P.S., Osbo n, T.J., 2003. The ole o he oceans in clima e. In . J.
Clima ol. 23, 1127–1159. h ps://doi.o g/10.1002/joc.926
B äge , S., Rome o Rod iguez, G.Q., Mulsow, S., 2020. The cu en s a us o en i onmen al
equi emen s o deep seabed mining issued by he In e na ional Seabed Au ho i y.
Ma . Policy, En i onmen al go e nance o deep seabed mining - scien ic insigh s and
ood o hough 114, 103258. h ps://doi.o g/10.1016/j.ma pol.2018.09.003
B own, A., Tha je, S., 2014. Explaining ba hyme ic di e si y pa e ns in ma ine ben hic
in e eb a es and deme sal shes: physiological con ibu ions o adap a ion o li e a
dep h. Biol. Re . 89, 406–426. h ps://doi.o g/10.1111/b .12061
Bu ka , N., Hube , M.F., 2021. A Su ey on he Explainabili y o Supe ised Machine
Lea ning. J. A i . In ell. Res. 70, 245–317. h ps://doi.o g/10.1613/jai .1.12228
Canonico, G., Bu igieg, P.L., Mon es, E., Mulle -Ka ge , F.E., S epien, C., W igh , D., Benson,
A., Helmu h, B., Cos ello, M., Sousa-Pin o, I., Saeedi, H., New on, J., Appel ans, W.,
Bedna šek, N., Bod ossy, L., Bes , B.D., B and , A., Goodwin, K.D., Iken, K., Ma ques,
A.C., Milosla ich, P., Os owski, M., Tu ne , W., Ach e be g, E.P., Ba y, T., De eo, O.,
Biga i, G., Hen y, L.-A., Rami o-Sánchez, B., Du án, P., Mo a o, T., Robe s, J.M.,
Ga cía-Aleg e, A., Cuad ado, M.S., Mu on, B., 2019. Global Obse a ional Needs and
Resou ces o Ma ine Biodi e si y. F on . Ma . Sci. 6.
h ps://doi.o g/10.3389/ ma s.2019.00367
Communi y ecology in he age o mul i a ia e mul iscale spa ial analysis - D ay - 2012 -
Ecological Monog aphs - Wiley Online Lib a y [WWW Documen ], n.d. URL
h ps://esajou nals.onlinelib a y.wiley.com/doi/ ull/10.1890/11-1183.1?casa_ oken=R
uXWXB_IzncAAAAA%3A8Auq90GX 5EnKgiilG-T_8uRaCeaX20TRgxK dJzy5eS-
BB1_slhzCzwoMLbdKyC784XOzOBPUUxdFUL (accessed 9.21.24).
Cudding on, K., Fo in, M.-J., Ge be , L.R., Has ings, A., Liebhold, A., O’Conno , M., Ray,
C., 2013. P ocess-based models a e equi ed o manage ecological sys ems in a changing
wo ld. Ecosphe e 4, a 20. h ps://doi.o g/10.1890/ES12-00178.1
66
Deng, J., Dong, W., Soche , R., Li, L.-J., Li, K., Fei-Fei, L., 2009. ImageNe : A la ge-scale
hie a chical image da abase, in: 2009 IEEE Con e ence on Compu e Vision and
Pa e n Recogni ion. P esen ed a he 2009 IEEE Con e ence on Compu e Vision and
Pa e n Recogni ion, pp. 248–255. h ps://doi.o g/10.1109/CVPR.2009.5206848
Du, J., 2018. Unde s anding o Objec De ec ion Based on CNN Family and YOLO. J. Phys.
Con . Se . 1004, 012029. h ps://doi.o g/10.1088/1742-6596/1004/1/012029
Du den, J.M., Schoening, T., Al haus, F., F iedman, A., Ga cia, R., Glo e , A.G., G eine , J.,
S ou , N.J., Jones, D.O.B., Jo d , A., Kaeli, J.W., Köse , K., Kuhnz, L.A., Lindsay, D.,
Mo is, K.J., Na kempe , T.W., Os e lo, J., Ruhl, H.A., Singh, H., Be , M.T.& B.J.,
2016. Pe spec i es in Visual Imaging o Ma ine Biology and Ecology: F om Acquisi ion
o Unde s anding, in: Oceanog aphy and Ma ine Biology. CRC P ess.
Eu opean Commission. Join Resea ch Cen e, In e na ional Council o he Explo a ion o
he Sea (ICES), 2010. Ma ine S a egy F amewo k Di ec i e : ask g oup 2 epo :
non-indigenous species, Ap il 2010. Publica ions Oce, LU.
E ans, D., 2006. The Habi a s o he Eu opean Union Habi a s Di ec i e. Biol. En i on. P oc.
R. I . Acad. 106B, 167–173.
E ans, J.L., Pecke , F., Howell, K.L., 2015. Combined applica ion o biophysical habi a
mapping and sys ema ic conse a ion planning o assess eciency and
ep esen a i eness o he exis ing High Seas MPA ne wo k in he No heas A lan ic.
ICES J. Ma . Sci. 72, 1483–1497. h ps://doi.o g/10.1093/icesjms/ s 012
Felden, J., Mölle , L., Schindle , U., Hube , R., Schumache , S., Koppe, R., Diepenb oek, M.,
Glöckne , F.O., 2023. PANGAEA - Da a Publishe o Ea h & En i onmen al Science.
Sci. Da a 10, 347. h ps://doi.o g/10.1038/s41597-023-02269-x
Fe ing, C., n.d. THE EUROPEAN GREEN DEAL.
Flanne y, E., P zeslawski, R., 2015. Compa ison o sampling me hods o assess ben hic ma ine
biodi e si y. A e spa ial and ecological ela ionships consis en among sampling gea ?
(Repo ). Geoscience Aus alia. h ps://doi.o g/10.11636/Reco d.2015.007
F ees one, A.L., Osman, R.W., Ruiz, G.M., To chin, M.E., 2011. S onge p eda ion in he
opics shapes species ichness pa e ns in ma ine communi ies. Ecology 92, 983–993.
h ps://doi.o g/10.1890/09-2379.1
F u os, I., Kaise , S., Pułaski, Ł., S udzian, M., Błażewicz, M., 2022. Challenges and Ad ances in
he Taxonomy o Deep-Sea Pe aca ida: F om T adi ional o Mode n Me hods. F on .
Ma . Sci. 9. h ps://doi.o g/10.3389/ ma s.2022.799191
Fu u e Science B ie s | Eu opean Ma ine Boa d [WWW Documen ], n.d. URL
h ps://www.ma ineboa d.eu/publica ion/ u u e-science-b ie (accessed 9.28.24).
Galpa so o, I., Bo ja, A., Uya a, M.C., 2014. Mapping ecosys em se ices p o ided by ben hic
67
habi a s in he Eu opean No h A lan ic Ocean. F on . Ma . Sci. 1.
h ps://doi.o g/10.3389/ ma s.2014.00023
Gam eld , L., Le check, J.S., By nes, J.E.K., Ca dinale, B.J., Duy, J.E., G in, J.N., 2015.
Ma ine biodi e si y and ecosys em unc ioning: wha ’s known and wha ’s nex ? Oikos
124, 252–265. h ps://doi.o g/10.1111/oik.01549
Glasby, G.P., 2002. Deep Seabed Mining: Pas Failu es and Fu u e P ospec s. Ma . Geo esou ces
Geo echnol. 20, 161–176. h ps://doi.o g/10.1080/03608860290051859
Guidi, L., Gue a, A.F., Canchaya, C., Cu y, E., Foglini, F., I isson, J.O., Malde, K., Ma shall,
C.T., Obs , M., Ribei o, R.P., Tjipu a, J., Heymans, S.J., Alexande , B., Piniella, Á.M.,
Kelle , P., Coopman, J., 2020. Big Da a in Ma ine Science. Eu opean Ma ine Boa d.
h ps://doi.o g/10.5281/zenodo.3755793
Hachaj, T., Mazu ek, P., 2020. Compa a i e Analysis o Supe ised and Unsupe ised
App oaches Applied o La ge-Scale “In The Wild” Face Ve ica ion. Symme y 12,
1832. h ps://doi.o g/10.3390/sym12111832
Haed ich, R.L., Rowe, G.T., Polloni, P.T., 1980. The megaben hic auna in he deep sea sou h o
New England, USA. Ma . Biol. 57, 165–179. h ps://doi.o g/10.1007/BF00390735
Ha is, P.T., Bake , E.K., 2012. 1 - Why Map Ben hic Habi a s?, in: Ha is, P.T., Bake , E.K.
(Eds.), Seaoo Geomo phology as Ben hic Habi a . Else ie , London, pp. 3–22.
h ps://doi.o g/10.1016/B978-0-12-385140-6.00001-3
Heg enæs, Ø., Gade, K., Hagen, O.K., Hagen, P.E., 2009. Unde wa e ansponde posi ioning
and na iga ion o au onomous unde wa e ehicles, in: OCEANS 2009. P esen ed a
he OCEANS 2009, pp. 1–7. h ps://doi.o g/10.23919/OCEANS.2009.5422358
Hu enne, V.A.I., Robe , K., Ma sh, L., Lo Iacono, C., Le Bas, T., Wynn, R.B., 2018. ROVs and
AUVs, in: Micalle , A., K as el, S., Sa ini, A. (Eds.), Subma ine Geomo phology.
Sp inge In e na ional Publishing, Cham, pp. 93–108.
h ps://doi.o g/10.1007/978-3-319-57852-1_7
Jain, S.M., 2022. Hugging Face, in: Jain, S.M. (Ed.), In oduc ion o T ans o me s o NLP:
Wi h he Hugging Face Lib a y and Models o Sol e P oblems. Ap ess, Be keley, CA,
pp. 51–67. h ps://doi.o g/10.1007/978-1-4842-8844-3_4
Lacha i é, M., Me axas, A., 2018. En i onmen al d i e s o epiben hic mega auna on a deep
empe a e con inen al shel : A mul iscale app oach. P og. Oceanog . 162, 171–186.
h ps://doi.o g/10.1016/j.pocean.2018.03.002
Langenkämpe , D., an Ke elae , R., Pu se , A., Na kempe , T.W., 2020. Gea -Induced
Concep D i in Ma ine Images and I s Eec on Deep Lea ning Classica ion. F on .
Ma . Sci. 7.
Langenkämpe , D., Zu owie z, M., Schoening, T., Na kempe , T.W., 2017. BIIGLE 2.0 -
68
B owsing and Anno a ing La ge Ma ine Image Collec ions. F on . Ma . Sci. 4, 83.
h ps://doi.o g/10.3389/ ma s.2017.00083
Legend e, P., Fo in, M.J., 1989. Spa ial pa e n and ecological analysis. Vege a io 80, 107–138.
h ps://doi.o g/10.1007/BF00048036
Le in, L.A., Be , B.J., Ga es, A.R., Heimbach, P., Howe, B.M., Janssen, F., McCu dy, A., Ruhl,
H.A., Snelg o e, P., S ocks, K.I., Bailey, D., Baumann-Picke ing, S., Bea e son, C.,
Beneld, M.C., Boo h, D.J., Ca ei o-Sil a, M., Colaço, A., Eblé, M.C., Fowle , A.M.,
Gje de, K.M., Jones, D.O.B., Ka suma a, K., Kelley, D., Le B is, N., Leona di, A.P.,
Lejze owicz, F., Mac eadie, P.I., McLean, D., Mei z, F., Mo a o, T., Ne bu n, A.,
Pawlowski, J., Smi h, C.R., Sun, S., Uchida, H., Va da o, M.F., Venka esan, R., Welle ,
R.A., 2019. Global Obse ing Needs in he Deep Ocean. F on . Ma . Sci. 6.
h ps://doi.o g/10.3389/ ma s.2019.00241
Li, J., T an, M., Siwabessy, J., 2016. Selec ing Op imal Random Fo es P edic i e Models: A
Case S udy on P edic ing he Spa ial Dis ibu ion o Seabed Ha dness. PLOS ONE 11,
e0149089. h ps://doi.o g/10.1371/jou nal.pone.0149089
Liu, F.T., Ting, K.M., Zhou, Z.-H., 2008. Isola ion Fo es , in: 2008 Eigh h IEEE In e na ional
Con e ence on Da a Mining. P esen ed a he 2008 Eigh h IEEE In e na ional
Con e ence on Da a Mining (ICDM), IEEE, Pisa, I aly, pp. 413–422.
h ps://doi.o g/10.1109/ICDM.2008.17
Lodge, M.W., 2011. In e na ional Seabed Au ho i y. In . J. Ma . Coas . Law 26, 463.
Lo ze, H.K., 2021. Ma ine biodi e si y conse a ion. Cu . Biol. 31, R1190–R1195.
h ps://doi.o g/10.1016/j.cub.2021.06.084
Lu, S., Ding, Y., Liu, M., Yin, Z., Yin, L., Zheng, W., 2023. Mul iscale Fea u e Ex ac ion and
Fusion o Image and Tex in VQA. In . J. Compu . In ell. Sys . 16, 54.
h ps://doi.o g/10.1007/s44196-023-00233-6
Luo, X., Qin, X., Wu, Z., Yang, F., Wang, M., Shang, J., 2019. Sedimen Classica ion o
Small-Size Seabed Acous ic Images Using Con olu ional Neu al Ne wo ks. IEEE Access
7, 98331–98339. h ps://doi.o g/10.1109/ACCESS.2019.2927366
Ma ín Míguez, B., No ellino, A., Vinci, M., Claus, S., Calewae , J.-B., Vallius, H., Schmi , T.,
Pi i o, A., Gio ge i, A., Askew, N., Iona, S., Schaap, D., Pina di, N., Ha pham, Q.,
Ka e , B.J., Populus, J., She, J., Palazo , A.V., McMeel, O., Ose , P., Lea , D., Manzella,
G.M.R., Go inge, P., Simoncelli, S., La kin, K., Holdswo h, N., A ani idis, C.D.,
Molina Jack, M.E., Cha es Mon e o, M. del M., He man, P.M.J., He nandez, F., 2019.
The Eu opean Ma ine Obse a ion and Da a Ne wo k (EMODne ): Visions and Roles
o he Ga eway o Ma ine Da a in Eu ope. F on . Ma . Sci. 6.
h ps://doi.o g/10.3389/ ma s.2019.00313
Maslej, N., Fa o ini, L., Pe aul , R., Pa li, V., Reuel, A., B ynjol sson, E., E chemendy, J.,
Lige , K., Lyons, T., Manyika, J., Niebles, J.C., Shoham, Y., Wald, R., Cla k, J., 2024.
69
A icial In elligence Index Repo 2024. h ps://doi.o g/10.48550/a Xi .2405.19522
Mbani, B., Buck, V., G eine , J., 2023a. An au oma ed image-based wo kow o de ec ing
megaben hic auna in op ical images wi h examples om he Cla ion–Clippe on Zone.
Sci. Rep. 13, 8350. h ps://doi.o g/10.1038/s41598-023-35518-5
Mbani, B., Schoening, T., Gazis, I.-Z., Koch, R., G eine , J., 2022a. Implemen a ion o an
au oma ed wo kow o image-based seaoo classica ion wi h examples om
manganese-nodule co e ed seabed a eas in he Cen al Pacic Ocean. Sci. Rep. 12,
15338. h ps://doi.o g/10.1038/s41598-022-19070-2
Mbani, B., Schoening, T., Gazis, I.-Z., Koch, R., G eine , J., 2022b. Implemen a ion o an
au oma ed wo kow o image-based seaoo classica ion wi h examples om
manganese-nodule co e ed seabed a eas in he Cen al Pacic Ocean. Sci. Rep. 12,
15338. h ps://doi.o g/10.1038/s41598-022-19070-2
Mbani, B., Schoening, T., G eine , J., 2023b. Au oma ed and In eg a ed Seaoo Classica ion
Wo kow (AI-SCW) [WWW Documen ]. h ps://doi.o g/10.3289/SW_2_2023
Mckinney, W., 2011. pandas: a Founda ional Py hon Lib a y o Da a Analysis and S a is ics.
Py hon High Pe o m. Sci. Compu .
Mika, S., Scholkop , B., Smola, A., n.d. Ke nel PCA and De-Noising in Fea u e Spaces 7.
Mon es, E., Le check, J.S., Gue a-Cas o, E., Klein, E., Ka anaugh, M.T., de Aze edo Mazzuco,
A.C., Biga i, G., Co dei o, C.A.M.M., Simoes, N., Macaya, E.C., Moi y, N.,
Londoño-C uz, E., Helmu h, B., Choi, F., So o, E.H., Milosla ich, P., Mulle -Ka ge ,
F.E., 2021. Op imizing La ge-Scale Biodi e si y Sampling Eo : Towa d an
Unbalanced Su ey Design. Oceanog aphy 34, 80–91.
Nimie -Da id, M., Vicini, D., Zel ne , T., Jakob, W., 2019. Mi suba 2: a e a ge able o wa d
and in e se ende e . ACM T ans G aph 38, 203:1-203:17.
h ps://doi.o g/10.1145/3355089.3356498
Oli e , T.H., Hea d, M.S., Isaac, N.J.B., Roy, D.B., P oc e , D., Eigenb od, F., F eckle on, R.,
Hec o , A., O me, C.D.L., Pe chey, O.L., P oença, V., Raaelli, D., Su le, K.B., Mace,
G.M., Ma ín-López, B., Woodcock, B.A., Bullock, J.M., 2015. Biodi e si y and
Resilience o Ecosys em Func ions. T ends Ecol. E ol. 30, 673–684.
h ps://doi.o g/10.1016/j. ee.2015.08.009
Pang, B., Nijkamp, E., Wu, Y.N., 2020. Deep Lea ning Wi h Tenso Flow: A Re iew. J. Educ.
Beha . S a . 45, 227–248. h ps://doi.o g/10.3102/1076998619872761
Paszke, A., G oss, S., Massa, F., Le e , A., B adbu y, J., Chanan, G., Killeen, T., Lin, Z.,
Gimelshein, N., An iga, L., Desmaison, A., Kop , A., Yang, E., DeVi o, Z., Raison, M.,
Tejani, A., Chilamku hy, S., S eine , B., Fang, L., Bai, J., Chin ala, S., 2019. PyTo ch:
An Impe a i e S yle, High-Pe o mance Deep Lea ning Lib a y, in: Ad ances in Neu al
In o ma ion P ocessing Sys ems. Cu an Associa es, Inc.
70
Payne, R.J., Egan, J., 2019. Using palaeoecological echniques o unde s and he impac s o pas
olcanic e up ions. Qua . In ., Dis al Eec s o Volcanic E up ions on P e-Indus ial
Socie ies 499, 278–289. h ps://doi.o g/10.1016/j.quain .2017.12.019
Pede sen, M., Hau um, J.B., Gade, R., Moeslund, T.B., 2019. De ec ion o Ma ine Animals in a
New Unde wa e Da ase wi h Va ying Visibili y 9.
Ped egosa, F., Va oquaux, G., G am o , A., Michel, V., Thi ion, B., G isel, O., Blondel, M.,
P e enho e , P., Weiss, R., Dubou g, V., Vande plas, J., Passos, A., Cou napeau, D.,
2011. Sciki -lea n: Machine Lea ning in Py hon. Mach. Lea n. PYTHON 6.
Peuke , A., Schoening, T., Ale izos, E., Köse , K., Kwasni schka, T., G eine , J., 2018.
Unde s anding Mn-nodule dis ibu ion and e alua ion o ela ed deep-sea mining
impac s using AUV-based hyd oacous ic and op ical da a. Biogeosciences 15,
2525–2549. h ps://doi.o g/10.5194/bg-15-2525-2018
Pu se , A., Ma con, Y., D eu e , S., Hoge, U., Sablo ny, B., Hehemann, L., Lembu g, J.,
Do schel, B., Biebow, H., Boe ius, A., 2019. Ocean Floo Obse a ion and Ba hyme y
Sys em (OFOBS): A New Towed Came a/Sona Sys em o Deep-Sea Habi a Su eys.
IEEE J. Ocean. Eng. 44, 87–99. h ps://doi.o g/10.1109/JOE.2018.2794095
Ren, S., He, K., Gi shick, R., Sun, J., 2015. Fas e R-CNN: Towa ds Real-Time Objec
De ec ion wi h Region P oposal Ne wo ks, in: Ad ances in Neu al In o ma ion
P ocessing Sys ems. Cu an Associa es, Inc.
Rigby, P., Piza o, O., Williams, S.B., 2006. Towa ds Geo-Re e enced AUV Na iga ion
Th ough Fusion o USBL and DVL Measu emen s, in: OCEANS 2006. P esen ed a
he OCEANS 2006, pp. 1–6. h ps://doi.o g/10.1109/OCEANS.2006.306898
Saeedi, H., Wa en, D., B and , A., 2022. The En i onmen al D i e s o Ben hic Fauna
Di e si y and Communi y Composi ion. F on . Ma . Sci. 9.
h ps://doi.o g/10.3389/ ma s.2022.804019
Sala, E., Knowl on, N., 2006. Global Ma ine Biodi e si y T ends. Annu. Re . En i on. Resou .
31, 93–122. h ps://doi.o g/10.1146/annu e .ene gy.31.020105.100235
Schoening, T., Jones, D.O.B., G eine , J., 2017. Compac -Mo phology-based poly-me allic
Nodule Delinea ion. Sci. Rep. 7, 13338. h ps://doi.o g/10.1038/s41598-017-13335-x
Schoening, T., Köse , K., G eine , J., 2018. An acquisi ion, cu a ion and managemen
wo kow o sus ainable, e aby e-scale ma ine image analysis. Sci. Da a 5, 180181.
h ps://doi.o g/10.1038/sda a.2018.181
Shin, H.-C., Ro h, H.R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollu a, D., Summe s,
R.M., 2016. Deep Con olu ional Neu al Ne wo ks o Compu e -Aided De ec ion:
CNN A chi ec u es, Da ase Cha ac e is ics and T ans e Lea ning. IEEE T ans. Med.
Imaging 35, 1285–1298. h ps://doi.o g/10.1109/TMI.2016.2528162
71
Singh, P.A., 2021. The wo-yea deadline o comple e he In e na ional Seabed Au ho i y’s
Mining Code: Key ou s anding ma e s ha s ill need o be esol ed. Ma . Policy 134,
104804. h ps://doi.o g/10.1016/j.ma pol.2021.104804
S a mann, T., Soe ae , K., Wei, C.-L., Lin, Y.-S., an Oe elen, D., 2019. The SCOC da abase, a
la ge, open, and global da abase wi h sedimen communi y oxygen consump ion a es.
Sci. Da a 6, 242. h ps://doi.o g/10.1038/s41597-019-0259-3
Szegedy, C., Vanhoucke, V., Ioe, S., Shlens, J., Wojna, Z., 2015. Re hinking he Incep ion
A chi ec u e o Compu e Vision. A Xi 151200567 Cs.
Tenhunen-Lunkka, A., Honkanen, R., 2024. P ojec coo dina ion success ac o s in Eu opean
Union- unded esea ch, de elopmen and inno a ion p ojec s unde he Ho izon 2020
and Ho izon Eu ope p og ammes. J. Inno . En ep. 13, 7.
h ps://doi.o g/10.1186/s13731-024-00363-x
T uby, J., B own, R.D., Ib ahim, I.A., Pa ellada, O.C., 2022. A Sandbox App oach o
Regula ing High-Risk A icial In elligence Applica ions. Eu . J. Risk Regul. 13,
270–294. h ps://doi.o g/10.1017/e .2021.52
an de Schoo , R., Depaoli, S., King, R., K ame , B., Mä ens, K., Tadesse, M.G., Vannucci, M.,
Gelman, A., Veen, D., Willemsen, J., Yau, C., 2021. Bayesian s a is ics and modelling.
Na . Re . Me hods P ime 1, 1–26. h ps://doi.o g/10.1038/s43586-020-00001-2
Vaudo, J.J., Hei haus, M.R., 2013. Mic ohabi a Selec ion by Ma ine Mesoconsume s in a
The mally He e ogeneous Habi a : Beha io al The mo egula ion o A oiding
P eda ion Risk? PLOS ONE 8, e61907. h ps://doi.o g/10.1371/jou nal.pone.0061907
Veale, M., Bo gesius, F.Z., 2021. Demys i ying he D a EU A icial In elligence Ac —
Analysing he good, he bad, and he unclea elemen s o he p oposed app oach.
Compu . Law Re . In . 22, 97–112. h ps://doi.o g/10.9785/c i-2021-220402
Volkmann, S.E., Lehnen, F., 2018. P oduc ion key gu es o planning he mining o
manganese nodules. Ma . Geo esou ces Geo echnol. 36, 360–375.
h ps://doi.o g/10.1080/1064119X.2017.1319448
Wang, C., Mei, D., Wang, Y., Yu, X., Sun, W., Wang, D., Chen, J., 2022. Task alloca ion o
Mul i-AUV sys em: A e iew. Ocean Eng. 266, 112911.
h ps://doi.o g/10.1016/j.oceaneng.2022.112911
Wa ne , G., 1982. Food and Feeding Mechanisms: Ophiu oidea, in: Echinode m Nu i ion.
CRC P ess.
Webb, T.J., Be ghe, E.V., O’Do , R., 2010. Biodi e si y’s Big We Sec e : The Global
Dis ibu ion o Ma ine Biological Reco ds Re eals Ch onic Unde -Explo a ion o he
Deep Pelagic Ocean. PLOS ONE 5, e10223.
h ps://doi.o g/10.1371/jou nal.pone.0010223
72
5
Vol.:(0123456789)
Scien i 㘶c Repo s | (2022) 12:15338 | h ps://doi.o g/10.1038/s41598-022-19070-2
www.na u e.com/scien i ic epo s/
shows he esul s o he same images a e colo no maliza ion by his og am ma ching he e e ence ECDF; we
poin ou ha hese colo no malized images ha e al eady been ans o med o a consis en spa ial esolu ion (in
pixel/cm), and hen cen e c opped o ep esen a s anda d spa ial oo p in o 1.6 squa e me e s on he seabed
(see de ails in “Image enhancemen s” sec ion). F om his igu e, he esul ing ECDF o he colo no malized
images a e iden ical o he e e ence dis ibu ion. Quali a i ely, his ans o ma ion esul s in no malized pho os
which ha e a good o e all scene b igh ness.
Sea 㘶oo classi 㘶ca ion assessmen . This sec ion p esen s esul s o he quan i a i e assessmen aimed
a e alua ing he pe o mance o bo h he supe ised and unsupe ised classi ie s on he es se images. Also
p o ided a e he esul s o he PCA p ojec ion o all classi ied images on o a wo-dimensional ea u e space,
which can be used o isualize how seman ically simila images g oup oge he in he ea u e space. Fo example,
whe eas images o la ge sized Mn-nodules g oup oge he on he uppe egion o his ea u e space, images o
densely dis ibu ed Mn-nodules g oup oge he on he le .
Pe o mance e alua ion o he ine- uned Incep ion V3 classi ie . The con usion ma ix used o e alua e he
pe o mance o he supe ised classi ie is shown in Fig.5A. The o -diagonal elemen s o he ma ix abula e
he numbe o misclassi ica ions made by he supe ised classi ie when i made p edic ions on he es images.
These es images comp ised 12.2% o he labeled images (612 in o al), since he o he 7% was used as alida-
ion se du ing hype -pa ame e s uning. The ma ix shows ha some o he es images labeled as Sea loo A
we e w ongly p edic ed o belong o Sea loo B. This could be because in some images o Sea loo A he sedi-
men blanke co e o e he Mn-nodules was no comple e (no high enough), and some ha we e only pa ly
Mn-nodules a e s ill isible; his caused hem o be misclassi ied as Sea loo B. The same easoning also explains
he con usion be ween Sea loo A and Sea loo D, depending on he gene al dis ibu ion o Mn-nodules (small
pa ches as in Sea loo B o homogenous dis ibu ion o la ge Mn-nodules (Sea loo D).
Figu e3. Compa ison be ween se s o (A) images co ec ed o illumina ion d op-o and, (B) adap i e
his og am equalized images. Also shown as inse s a e he in ensi y his og ams showing he dis ibu ion o he
pixel alues o each RGB channel.
6
Vol:.(1234567890)
Scien i 㘶c Repo s | (2022) 12:15338 | h ps://doi.o g/10.1038/s41598-022-19070-2
www.na u e.com/scien i ic epo s/
Figu e5B shows he de elopmen o he lea ning cu es used o ack he p og ess o model aining a e
e e y i e a ion s ep. The shape o he c oss-en opy loss cu e dec eases wi h e e y i e a ion, and con e ges wi h
a inal loss alue o 0.09. On he o he hand, he shape o he accu acy cu e ises s eadily wi h each i e a ion
un il model con e gence.
F1 sco e was used as he accu acy me ic o e alua e he pe o mance o he ine- uned Incep ion V3 classi ie ;
i p o ides he ha monic mean o p ecision and ecall, and anges be ween a wo s sco e o 0 o he bes sco e
alue o 1. Using his app oach, he F1 sco e o ou ine- uned classi ie was ound o be 0.93.
Figu e4. (A) Re e ence image used o colo no maliza ion and, (B) The Empi ical Cumula i e Dis ibu ion
Func ion o he e e ence image. (C) Compa ison be ween adap i e his og am equalized images and, (D) colo
no malized and escaled cen e c opped images.
7
Vol.:(0123456789)
Scien i 㘶c Repo s | (2022) 12:15338 | h ps://doi.o g/10.1038/s41598-022-19070-2
www.na u e.com/scien i ic epo s/
Figu e5C shows he dis ibu ion o p edic ion con idence sco es o each o he sea loo classes. The p edic-
ion con idence was g ea e han 0.6 in mo e han 80% o he images, o all ou classes. Figu e5D shows h ee
example images o each o he ou sea loo classes as p edic ed by he ine- uned Incep ion V3 classi ie .
Pe o mance e alua ion o sampling s a egies used in i ing k-means classi ie s. Figu e6 shows he esul s o
he compa ison o he ou aining se sampling s a egies. As men ioned, each s a egy was e alua ed based on
he ime i ook om gene a ing he aining images o i ing a k-means classi ie , and also on he quali y o he
esul ing clus e s. Random sampling was he quickes wi h a ime- o- i o 0.12s. S a i ied clus e -based sam-
pling esul ed in he bes quali y o clus e s, wi h an absolu e silhoue e sco e o 0.4. Spa ially uni o m sampling
was he slowes wi h a ime- o- i o 7s, despi e he quali y o i s clus e s being equal o bo h andom sampling
and p obabilis ic weigh ed esampling. The op imal sampling s a egy was chosen as s a i ied clus e -based
sampling. This is because i achie ed he highes silhoue e sco e o 0.4, and was only 1.7s slowe han he second
as es s a egy o he p obabilis ic esampling.
T ea ing he Incep ion V3 classi ica ion esul s as g ound u hs, he Fowlkes-Mallows Index (FMI) was used
o quan i y he success o ou k-means classi ie in success ully de ining clus e s ha a e simila o he g ound
u h se o classes. The FMI is he geome ic mean o he p ecision and ecall ha makes no assump ion abou
he clus e s uc u e29,30. Using his app oach, we ob ained an FMI sco e o 0.5, which indica es good simila i y
in he classi ica ion accu acies.
Figu e7 shows he PCA p ojec ion o all he images colo coded by he esul s o bo h unsupe ised and
supe ised classi ica ion. The class bounda ies o he unsupe ised classi ie a e ab up and well-de ined. On
he o he hand, he supe ised classi ie esul s in uzzy class bounda ies, which is he case in eali y since he
ansi ion be ween classes is sub le. O e all, he wo classi ie s gene a e esul s ha ag ee wi h a Cohen’s kappa
coe icien o 0.6.
Figu e8 shows he PCA p ojec ion o all he images wi hou colo coding, in which seman ically simila
images (e.g., hose wi h la ge Mn-nodules) can be seen o g oup oge he in he ea u e space.
Spa ial dis ibu ion o sea 㘶oo classes. This sec ion pu s he image-based classi ica ion esul s in o
geog aphic pe spec i e, by mapping ou he dis ibu ion o he sea loo classes spa ially wi hin bo h he Ge man
and Belgian wo king a eas.
Figu e5. (A) Con usion ma ix o supe ised classi ie pe o mance e alua ion. (B) Loss and accu acy cu es
used o moni o p og ess o model aining. (C) Dis ibu ion o p edic ion con idence sco es o each sea loo
class. (D) Examples o classi ie p edic ions o each sea loo class.
8
Vol:.(1234567890)
Scien i 㘶c Repo s | (2022) 12:15338 | h ps://doi.o g/10.1038/s41598-022-19070-2
www.na u e.com/scien i ic epo s/
Figu e6. Pe o mance e alua ion o unsupe ised classi ie s i using aining da a gene a ed using each
sampling s a egies.
Figu e7. PCA p ojec ion o all images on o ea u e space colo coded by esul s o (A) unsupe ised
classi ica ion and, (B) supe ised classi ica ion.
9
Vol.:(0123456789)
Scien i 㘶c Repo s | (2022) 12:15338 | h ps://doi.o g/10.1038/s41598-022-19070-2
www.na u e.com/scien i ic epo s/
The spa ial dis ibu ion o he sea loo classes o bo h Ge man and Belgian con ac a eas is shown in Fig.9,
by colo -coding he image loca ions along he deploymen acks based on he supe ised Incep ion V3 clas-
si ica ion esul s. The no he n hal o he Belgian a ea comp ises dense nodule co e ed sea loo (Sea loo C),
in e mingled wi h pa ches o la ge nodules o Sea loo D a local ele a ions. Sea loo D becomes he dominan
class in he dep essions in he no he n pa o he a ea, and along he wo acks in he sou he n pa . Ve y ew
ins ances o Sea loo class A and B exis in he Belgian a ea.
The Ge man con ac a ea ypically comp ises Sea loo B in he no he n pa , and a mix o Sea loo B and D
in he sou he n pa . The Ge man a ea does no con ain a signi ican amoun o Sea loo C, highligh ing ha he
Mn-nodules a e gene ally smalle compa ed o he Belgian a ea. The occu ences o ins ances o Sea loo A a e
clea ly co ela ed o he loca ions o he d edge expe imen conduc ed in he Ge man a ea. In Fig.10, p e- and
pos -d edge deploymen acks a e shown, along wi h example images. A e he d edge expe imen , he p opo -
ion o Sea loo A inc eased by o e 30%, caused by he sedimen u no e /ploughing and subsequen se ling
o he suspended sedimen on o Mn-nodules a e a ce ain ime (see also31). In o de o allow easy isualiza ion
o he impac o he d edge expe imen , only he pos -d edge ack ha has a co esponding p e-d edge ack is
shown in Fig.10. The o he wo pos -d edge acks also i wi h he occu ences om Sea loo A.
Discussion
Fo s uc u ing he discussion o cla i y, we b ie ly summa ize wha has been p esen ed abo e, and make ele an
connec ions o a ious aspec s. In gene al, i can be said ha ecen de elopmen s in bo h unde wa e imaging
echnologies and hyd o-acous ics ha e allowed ma ine esea che s o cha ac e ize and g ound- u h seabed
subs a e classes32–34. In his pape , we p esen an Au oma ed and In eg a ed Sea loo Classi ica ion Wo k low
(AI-SCW), which includes a module ha au oma ically de ec s lase poin s om images, and uses hem o scale
de e mina ion. This is hen used o co ec o illumina ion a e ac s, and o colo no malize images eco ded
a di e en imes du ing di e en came a deploymen s. This gene a es a isually homogenous image da a se
ha can be used as aining da ase s o he au oma ed classi ica ion. To his end, he wo k low also includes a
module o semi-au oma ic labeling, which educes he manual e o equi ed o anno a e images needed o
aining classi ica ion models. Bo h supe ised Incep ion V3 and unsupe ised k-means classi ie s ha e been
ained, and hei classi ica ion esul s compa ed. The Incep ion V3 classi ie is ained using semi-au oma ically
gene a ed labels, while he k-means classi ie is ained using a subse o unlabeled images ( hose ha e been gen-
e a ed using a s a i ied clus e -based sampling s a egy). When he esul s o bo h he Incep ion V3 and k-means
classi ie s we e compa ed, hey showed a good ag eemen , wi h a Cohen’s kappa coe icien o 0.6. Below, he
a ious aspec s o AI-SCW wo k low a e i s discussed in de ail. Finally, we discuss he spa ial dis ibu ion o
he de i ed sea loo classes in he con ex o e ain and backsca e p ope ies o he sea loo .
As pa o he lase poin de ec ion wo k low, he sub ac ion o he ed channel om he linea combina ion
o blue and g een channels wo ked e y well, p oducing an in e media e image wi h e y high alues a ound
he lase poin s and e y low alues elsewhe e; he in ensi y maxima o his in e media e image con ained he
h ee lase s. We obse ed ha he linea combina ion had o be scaled by a coe icien o educe he e ec o he
co ela ion among he RGB channels. A e some i e a ions, we ound ha coe icien alues be ween 0 and 1
p oduced good esul s ( ue posi i es), and in pa icula , a coe icien alue o 0.2 p oduced he bes esul s o
his da ase . This alue may di e when he wo k low is applied in o he da ase s e.g., depending on he ligh -
ing condi ion, dep h, ype o de ice eco ding he images e c. We u he obse ed ha de ec ions based on he
con as enhanced images esul ed in many alse lase spo de ec ions. This is because he con as enhancemen
ans o ma ion educed he in ensi y o he ed lase poin s, by adap i ely equalizing he dis ibu ion o in ensi y
alues in local image iles. The lase poin de ec ion accu acy inc eased signi ican ly when he de ec ion was
done on he o iginal pho os. This is because he lase s a e a ge ed owa ds he cen e o he ield o iew o he
Figu e8. PCA p ojec ion o all images on o ea u e space wi hou colo coding. Seman ically simila images
(e.g., hose wi h la ge Mn-nodules) can be seen o g oup oge he .
10
Vol:.(1234567890)
Scien i 㘶c Repo s | (2022) 12:15338 | h ps://doi.o g/10.1038/s41598-022-19070-2
www.na u e.com/scien i ic epo s/
came a, a egion ha was al eady well illumina ed by he a i icial ligh sou ce; which made he ed lase poin s
p ominen and easy o de ec . In addi ion, since he mask used o il e he in ensi y maxima coo dina es was he
same o all images, i allowed us o implemen he lase poin de ec ion wo k low concu en ly ac oss all CPU
co es. This inc eased he p ocessing speed by up o 3 imes, compa ed o a sequen ial implemen a ion. When
compa ed o he DELPHI sys em implemen ed in a p e ious s udy5, ou implemen a ion elies on manually hand
labeled in o ma ion om jus one single example image, compa ed o he DELPHI sys em ha i e a i ely uses
70 manually anno a ed images. As ou da ase speci ically comp ised h ee lase poin s p ojec ed om a came a
al i ude o be ween 1 o 4m, u u e esea ch could imp o e his lase poin de ec ion wo k low o accommoda e
o he possible scena ios e.g., whe e he e a e ewe han h ee lase poin s isible in he image.
Ou semi-au oma ed app oach o gene a ing he labeled aining se o be used o ine uning he supe ised
Incep ion V3 classi ie was bo h s aigh o wa d and quick o execu e. The semi-au oma ion in ol ed some
manual anno a ion o example images by a human analys , ollowed by an au oma ed nea es neighbo sampling
o addi ional aining images in he ea u e space. This se -up was used o explici ly include domain knowledge
du ing he labeled aining da a gene a ion p ocess. Inco po a ion o domain knowledge was impo an because
nea es neighbo sampling only makes sense when seman ically simila images a e mapped close oge he in
ea u e space. In ou case, domain knowledge was inco po a ed by encoding ex u e and en opy in o he ea u e
Figu e9. Maps o deploymen acks in bo h he Ge man (BGR) and Belgian (GSR) con ac a eas. The
deploymen acks a e colo coded by supe ised classi ica ion esul s, and o e laid on ou g idded mul ibeam
ba hyme ic da ase . The bounda ies o polyme allic nodule explo a ion a eas we e sou ced om In e na ional
Seabed Au ho i y (h ps:// www. isa. o g. jm), while he base map in he o e iew map was sou ced om GEBCO
(h ps:// www. gebco. ne /).
11
Vol.:(0123456789)
Scien i 㘶c Repo s | (2022) 12:15338 | h ps://doi.o g/10.1038/s41598-022-19070-2
www.na u e.com/scien i ic epo s/
ec o s ha ep esen he images in ea u e space, because he seabed in ou wo king a ea was mos ly cha ac-
e ized by Mn-nodules ha appea as i egula blobs o black pixels in he image. The use o ex u e ea u es o
encode seabed cha ac e is ics has been epo ed in p e ious s udies, which ound ha hey adequa ely e eal
sub le a ia ion in seabed cha ac e is ics19–21. In addi ion, domain knowledge was inco po a ed by in e ac i ely
choosing example images ha a e uni o mly dis ibu ed in he wo-dimensional PCA-p ojec ed ea u e space;
his educed he class imbalance in he sample aining se . We emphasize ha he abo e-desc ibed ea u e
space was only used o gene a ing labeled aining da a, a e which deep lea ning was employed o supe ised
classi ica ion. Focusing on supe ised classi ica ion, we obse ed ha he Incep ion V3 model, which was ine-
uned using he semi-au oma ically gene a ed labeled da ase achie ed a good classi ica ion pe o mance. This
can be a ibu ed o he abili y o deep lea ning a chi ec u es o ex ac use ul abs ac ea u es du ing aining,
as poin ed ou e.g. by35. Achie ing such a good pe o mance using a hand ul o human-labeled examples pa ly
add esses he bo leneck o manual anno a ion o la ge da a se s. The e o e, ou app oach con ibu es owa ds
making he adop ion o s a e-o - he a compu e ision models in o image based ma ine s udies much easie .
When explo ing he esul s o he unsupe ised classi ica ion in he PCA-p ojec ed ea u e space, we obse ed
ha he in e -clus e sepa a ion dis ance was small. This obse a ion is ypical o images o he deep-sea, whe e
he a ia ion in seabed cha ac e is ics is sub le o e kilome e scale, as was also poin ed ou in p e ious s udies
such as36. Ou compa ison o he ou s a egies o selec ing he app op ia e aining da a o i ing an unsupe -
ised classi ie showed ha wi h such images, andom sampling may be he ideal app oach o gene a ing his
aining da a se , i quick compu a ion is mos impo an . Despi e he speed, howe e , andom sampling may
cause imbalance in he aining da a, since samples will be disp opo iona ely d awn om egions o he ea u e
space ha a e o e - ep esen ed. This is likely o occu e.g., when images a e collec ed in he la gely homogenous
abyssal Mn-nodule plains wi h only in equen egions o no (sedimen co e ed) nodules o o he a e seabed
cha ac e is ics. In a p e ious s udy, P a i e al.37 also co obo a e his ela ionship be ween andom sampling
and class imbalance. Fu he mo e, aining a classi ie using imbalanced da a may esul in poo gene aliza ion
pe o mance, as was also poin ed ou by38. When he me ic o in e es is no speed bu quali y o he esul ing
clus e s, ou expe imen showed ha s a i ied clus e -based sampling is a be e app oach. This becomes e iden
Figu e10. Deploymen acks in he Ge man con ac a ea (BGR), which show he sea loo classi ica ion
esul s be o e and a e he d edge expe imen s. The d edge expe imen inc eased he p opo ion o sedimen -
co e ed seabed (Sea loo A) by 30%. Fo con ex , he ‘be o e’ ack is shown in g ay o e laid as backg ound in
he bo om panel. Fo cla i y, only one obse a ion ack is shown om a e he d edge expe imen .
12
Vol:.(1234567890)
Scien i 㘶c Repo s | (2022) 12:15338 | h ps://doi.o g/10.1038/s41598-022-19070-2
www.na u e.com/scien i ic epo s/
as i esul ed in he highes absolu e silhoue e sco e, while being only 2s slowe han andom sampling. The high
silhoue e sco e implies ha o all he o he me hods we compa ed, he s a i ied clus e -based sampling app oach
esul ed in clus e s ha a e clea ly dis inguished om each o he . This is because he app oach inco po a es an
o e -clus e ing s ep ha i s pa i ions he en i e en opy-de ined ea u e space, which allows samples o be
d awn uni o mly ac oss all egions o he ea u e space, na u ally educing class imbalance. Da a om a mo e
he e ogenous sea loo like in shallow wa e en i onmen s may no exhibi a simila ea u e space dis ibu ion,
and he e o e a di e en sampling s a egy may be equi ed ha needs o be es ed in u u e s udies.
Visualizing he unsupe ised sea loo classi ica ion esul s o he k-means clus e ing in ea u e space showed
ab up in e -class bounda ies. This is an a i ac o he k-means objec i e unc ion ha esul s in con ex clus e s
wi h well-de ined bounda ies, as was also poin ed ou in a p e ious s udy by39. In he li e a u e, he e ha e been
a emp s o explici ly in oduce uzzy class bounda ies in sea loo classi ica ion e.g.40. O he s such as41 a emp ed
o cus omize k-means by modi ying i s objec i e unc ion using leas -squa es c i e ion o implici ly in oduce
uzziness. Howe e , he au ho s clea ly poin ou ha hese uzzy clus e ing app oaches a e e y sensi i e o clus-
e size imbalance, and can esul in clus e s whose seman ic meaning is ha d o in e p e . In ou supe ised case
o he Incep ion V3, he bounda ies we e logically uzzy in he PCA-p ojec ed space because he con olu ional
neu al ne wo k ou pu s a se o p obabili ies. This allows he model o make so classi ica ion by assigning each
image a likelihood o belonging o each o he classes using a con idence sco e. Despi e hese gene al di e ences,
ou quan i a i e assessmen o bo h he supe ised Incep ion V3 and unsupe ised k-means classi ica ion esul s
indica e a good ag eemen , wi h a Cohen’s kappa coe icien o 0.6. McHugh42 also quan i ied he ex en o which
wo di e en classi ie s assign he same sco e o a a iable using he Cohen’s kappa s a is ic. F om hese assess-
men s, i can be concluded ha a as , p elimina y sea loo classi ica ion while a sea may be pe o med wi h
unsupe ised me hods. The main ad an age o using k-means is he low compu ing powe needed, which ne e -
heless p oduces easonable classi ica ion esul s. Supe ised me hods which equi e GPU compu ing esou ces
may be be e execu ed o sea loo classi ica ions as pa o a de ailed habi a cha ac e iza ion in a second s ep.
They a e be e sui ed o pick ou sub le ansi ions be ween classes, which is usually he case in eali y.
Rega ding he geospa ial dis ibu ion o sea loo classes and hei co ela ion wi h ba hyme y, he isu-
aliza ion o ou sea loo classes in map iew makes i ob ious ha hei spa ial dis ibu ion is no andom/
homogenous bu clus e ed. In he Belgian a ea, he dense Mn-nodules (Sea loo C) and la ge sized Mn-nodules
(Sea loo D) occupy 94% o he OFOS-inspec ed a ea a abou 4500m wa e dep h. The Ge man con ac a ea
p e-dominan ly con ains spa sely dis ibu ed Mn-nodules (Sea loo B) in he No h, while he sou h comp ises a
mix u e o bo h Sea loo B and Sea loo D. O e all, 96% o spa sely dis ibu ed Mn-nodules make up he sea loo
in he Ge man a ea a wa e dep hs o app oxima ely 4100m. The sea loo class D wi h big Mn-nodules lies in
deepe egions o he sea loo wi h ugged e ain, whe eas spa sely dis ibu ed nodules occupy shallowe egions
o he sea loo wi h la e ain (please see u he de ails in he supplemen a y in o ma ion). This is consis en
wi h he indings o p io s udies, which showed ha a highe numbe o Mn-nodules occu in a eas wi h a ug-
ged sea loo han la plains e.g.,41,43,44. A he si e o he d edge expe imen , s e ches o sedimen co e ed, o
d edged/ploughed seabed (Sea loo A) appea s a e he expe imen ; he p opo ion o sedimen co e ed seabed
inc eased by 30% (see Fig.10).
Compa ing ou app oaches o ela ed wo ks, ou colo no maliza ion app oach based on au oma ically
de ec ed lase poin s in oduced a simple ye no el wo k low, which signi ican ly educed he unde wa e illu-
mina ion a i ac s on ou images. Simila o p e ious wo ks e.g. by24, ou app oach escaled and no malized he
colo o each image depending on i s al i ude abo e he sea loo : images eco ded a he om he sea loo we e
ans o med mo e han hose close o he sea loo . In con as , howe e , ou app oach was di e en since we did
no ha e access o he al i ude o each image, and he e o e we implemen ed a no el app oach ha in e s hem
om au oma ically de ec ed lase poin s. Fu he mo e, ou app oach uses his og am ma ching o co ec o
une en scene b igh ness among he pho os, which is simple, pa allelizable, and does no e en equi e knowledge
o pa ame e s equi ed o econs uc ion o he pa h o ligh ays h ough he wa e column e.g. as demanded
by physics-based app oaches24,45,46, o pho og amme ic s uc u e om-mo ion47 and simul aneous localiza ion
and mapping (SLAM)48,49. Mo eo e , pe o ming colo no maliza ion on he aw images imp o ed he accu acy
o sea loo classi ica ion; his is simila o obse a ions by p e ious wo ks such as by24,45. Pa icula ly in ou case,
he aw images acqui ed a a ying al i ude we e no di ec ly compa able; hese images ep esen egions o he
sea loo wi h a ying spa ial oo p in and scene b igh ness. Wi h espec o gene a ing anno a ions o aining
classi ie , ou semi-au oma ed labeling app oach g ea ly educes human e o simila o p e ious wo ks by11,50,51.
Howe e , ou app oach is no el because i allows o he inco po a ion o domain knowledge h ough ea u e
space enginee ing, a he han only elying on simila i y in geog aphic space and spa ial au o co ela ion e.g. as
p oposed by52. Rega ding op ical image-based sea loo classi ica ion, ou esul s e ealed seabed subs a e classes
ha had seman ic meaning, simila o p e ious wo ks by20,21,24,25,53,54. Howe e , ou wo k low is no el since we
implemen bo h supe ised (Incep ion V3) and unsupe ised (k-means) classi ie s, and we u he demons a e
ha bo h o hem show a good o e all ag eemen in sea loo classi ica ion accu acy. Thus, ou s udy p o ides an
unsupe ised sea loo classi ica ion wo k low ha can be used a sea whe e compu e s a e no e y powe ul, as
well a supe ised wo k low ha is sui able o o ice se ings whe e he e is access o compu e s wi h inc eased
memo y and GPU ha dwa e.
Conclusion and ecommenda ions o u u e esea ch
This s udy con ibu es o he cu en unde s anding o image-based deep seabed classi ica ion, and i s no el y
can be summa ized as ollows: Fi s , we implemen a new app oach ha au oma ically de ec s lase poin s om
a sequence o unde wa e images. This is use ul o calcula ing scale, which allows ma ine esea che s who make
measu emen s on he images o con e om pixel uni s o eal wo ld me ic uni s e.g., me e s. The scale also
13
Vol.:(0123456789)
Scien i 㘶c Repo s | (2022) 12:15338 | h ps://doi.o g/10.1038/s41598-022-19070-2
www.na u e.com/scien i ic epo s/
allows he scien is s o in e he heigh abo e he sea loo om which he image was acqui ed, which is use ul o
de e mining he spa ial oo p in o he image; Second, we implemen a semi-au oma ed app oach ha d as i-
cally educes he equi ed human e o du ing image labeling. This is use ul o ma ine scien is s who ou inely
wo k wi h la ge olumes o unde wa e images, since i educes he bu den and a igue o gene a ing manual
anno a ions, while s ill allowing hem o ain high-accu acy classi ica ion models (e.g., con olu ional neu al
ne wo ks, andom o es s e c.) ha au oma ically analyze he images and subsequen ly classi y he sea loo in o
habi a ypes; Thi d, we p opose s a i ied clus e -based sampling as good s a egy o gene a ing a subse o
images o be used o aining an unsupe ised classi ie . When aced wi h huge olumes o images ollowing an
expedi ion, his p oposed sampling s a egy is use ul o help ma ine scien is s when deciding how o gene a e a
aining se ha i s in s anda d compu e memo y, while simul aneously being class balanced and no equi ing
specialized ha dwa e such as GPU.
Al hough we p opose s a i ied clus e -based sampling as he op imal unsupe ised classi ie , his conclu-
sion is speci ic o unde wa e pho os eco ded in he deep-sea abyssal plains, o which he sea loo is mos ly
homogenous. This may no hold in shallowe dep hs wi h he e ogenous sea loo cha ac e is ics, and u u e
esea ch could add ess his by ex ending ou AI-SCW ea u e ex ac ion module o in es iga e e.g., i colo
di e ences could be use ul in classi ying sea loo co e ed by o he subs a e classes such as seag ass, ocks and
sand. In addi ion, u u e esea ch could explo e he possibili y o embedding au oma ed image-based wo k lows
such as AI-SCW on o he da a acquisi ion so wa e o ROV/UAV. This would enable eal ime on- he- ly image
analysis, which p o ides signi ican sa ings in ime and pos p ocessing e o o applica ions such as seabed
classi ica ion, megaben hic auna de ec ion as well as he quan i ica ion o c us s and nodules; hese a e ongoing
de elopmen s a GEOMAR and o he cen e s.
As a concluding ema k, his s udy comp ehensi ely p esen s a se o me hods ha oge he o m an au o-
ma ed wo k low o image-based sea loo classi ica ion and mapping. These include: au oma ed lase poin
de ec ion; semi-au oma ed image anno a ion; as well as supe ised and unsupe ised sea loo classi ica ion.
The applicabili y o hese me hods is demons a ed using a case s udy in ol ing sea loo images om Mn-
nodule co e ed deep seabed a eas o he Cla ion Clippe on Zone in he Cen al Paci ic Ocean. In so doing, we
clea ly demons a e po en ial ways o inco po a ing ecen ad ances in machine lea ning and compu e ision
in o ma ine esea ch, especially o pu poses o gene a ing ac ionable insigh s om he huge olumes o op i-
cal unde wa e image y, which a e nowadays eco ded du ing scien i ic expedi ions by imaging sys ems such
as AUV, ROV and OFOS. We belie e ha hese insigh s signi ican ly con ibu e owa ds he b oade aim o
unde s anding ou ma ine ecosys ems, which in u n enables app op ia e measu es o be es ablished o hei
managemen and sus ainable use.
Ma e ials and me hods
Wo king a ea. As a case s udy, AI-SCW was applied o an unde wa e image da ase eco ded du ing an
expedi ion o he Ge man and Belgian con ac a eas o manganese nodule (Mn-nodule) mining, a he Cla ion
Clippe on Zone (CCZ) in he cen al Paci ic Ocean. The expedi ion was pa o he second phase o he JPI-
Oceans p ojec MiningImpac , and was execu ed on boa d he Ge man esea ch essel SONNE du ing c uise
SO268. The p ojec aimed a assessing how po en ial mining o polyme allic nodules on he sea loo would
impac he deep-sea en i onmen . A o al o 12 ideo in es iga ions we e pe o med in wo di e en con ac
a eas (Table1). Wi hin he Ge man con ac a ea, a small-scale sedimen plume expe imen was conduc ed,
using a chain d edge o obse e he e-deposi ioning o plume sedimen s be o e and a e he dis u bance. Th ee
came a deploymen s we e conduc ed o pho og aph he sea loo a e he d edge expe imen , while one deploy-
men was conduc ed be o e he expe imen o compa ison55.
Se up o he image acquisi ion sys em. The unde wa e images we e acqui ed using an Ocean Floo Obse a-
ion Sys em (OFOS), which was owed a a speed o 0.5 kno s a 1 o 4m abo e he sea loo . I comp ised a s eel
ame equipped wi h bo h s ill and ideo came as. The s ill images we e eco ded using a Canon EOS 5D Ma k
IV came a wi h a 24mm lens, whe eas ideo was eco ded using he HD-SDI came a wi h 64° × 40° iew. These
wo came as we e spaced 50cm apa and di ec ed e ically owa ds he sea loo alongside wo s obe ligh s
(Sea&Sea YS-250), ou LED ligh s (SeaLi e Sphe e), h ee lase s spaced 40cm apa , one al ime e and one
USBL sys em o acking he posi ion o he OFOS. Whe eas he ideo came a eco ded con inuously, he s ill
came a ook an image once e e y 10s. Bo h came as had a dome po ha did no al e he ield o iew as long
as he lens was cen e ed p ope ly wi hin he dome. Came a calib a ion was done by pho og aphing a came a
calib a ion a ge on deck be o e he ac ual deploymen . Fu he de ails ega ding he image acquisi ion se up
can be ound on page 65 o he SO268 c uise epo 55.
Image da ase . The image da ase comp ised 40,678 unde wa e s ill images, which we e eco ded du ing he
12 deploymen s o he owed Ocean Floo Obse a ion Sys em (OFOS). The espec i e da a a e published on
PANGAEA56, and can be accessed online (h ps:// doi. panga ea. de/ 10. 1594/ PANGA EA. 935856). In addi ion o
he PANGAEA da ase , he images can also be accessed upon eques as se ices h ough he BIIGLE po al
(h ps:// anno a e. geoma . de/ p oje c s/ 44). BIIGLE is an online image anno a ion pla o m, speci ically de eloped
o acili a e he anno a ion o ben hic auna om unde wa e images57. The OFOS deploymen s we e conduc ed
in an a e age wa e dep h o 4,280m, co e ing a ack leng h o 92.5km in o al. A e acquisi ion, he images
we e geo e e enced by ma ching each image’s acquisi ion ime in UTC o he USBL na iga ion da a. Th ee lase
poin e s in a iangula con igu a ion a e used o scaling. Ligh is p o ided h ough se e al ligh s ocusing on
he cen al a ea below he OFOS ame illumina ing he ield o iew o he e ically downwa d looking came a.
14
Vol:.(1234567890)
Scien i 㘶c Repo s | (2022) 12:15338 | h ps://doi.o g/10.1038/s41598-022-19070-2
www.na u e.com/scien i ic epo s/
De ailed in o ma ion abou hese images and hei acquisi ion can be ound in he SO268 c uise epo 55. Table1
p o ides an o e iew o he OFOS deploymen s du ing he expedi ion.
So wa e and APIs. AI-SCW has been implemen ed using he Py hon p og amming language. Some o
he majo lib a ies used include sciki -image, sciki -lea n, Tenso Flow and pandas. Supplemen a y TableS1 p o-
ides a comple e lis o all he speci ic py hon lib a ies used in AI-SCW as well as a b ie desc ip ion o hei use.
In addi ion, he speci ic py hon sc ip s used in implemen ing each componen o AI-SCW is shown in supple-
men a y TableS2. All o hese sc ip s can be ound online in his public Gi lab eposi o y (h ps:// gi . geoma . de/
open- sou ce/ AI- SCW), whe e he comple e sou ce code iles o he AI-SCW p ojec a e open sou ce. Alongside
he sou ce code iles, a de ailed guide o se ing up he p og amming en i onmen and execu ing he espec i e
sc ip s o each componen o AI-SCW is p o ided.
Image enhancemen s. Scale de e mina ion by lase poin de ec ion. Th ee lase poin s p ojec ed on o he
sea loo and pho og aphed in each image a e used o de e mine he image scale. This scale is use ul o ma ine
scien is s who ely on measu emen s made on he images, since i o ms he basis o he con e sion om pixel
uni s o eal wo ld uni s. The scale is calcula ed as a a io be ween he dis ance sepa a ing he h ee lase poin s
(in pixel uni s) and hei calib a ed dis ance measu ed in eal wo ld uni s (cen ime e s). Manually anno a ing
he lase poin s om housands o images is labo ious and non-scalable. This p o ides he p ima y mo i a ion
o au oma ing he lase poin de ec ion. Below, we desc ibe an app oach o au oma ically de ec ing lase s om
each image, and using hese de ec ions o calcula e he scale o each co esponding image.
Each image I o he da a se consis s o h ee-colo channels (I(R), I(G), I(B)) and each o hese channels has
a pixel wid h w = 4480 and pixel heigh h = 6720. One aining image wi h well isible lase poin s is manually
selec ed and anno a ed. The anno a ions p o ide an es ima e o he pixel coo dina es o lase poin s in all images.
This is done by c ea ing a mask Mlp as iangle which connec s he h ee anno a ed lase poin s. To allow o a i-
abili y in lase poin coo dina es caused by a ying OFOS al i ude, a bu e is added a ound Mlp. This bu e is
chosen as 250 pixels in ou implemen a ion. I was de e mined by andomly checking di e en images o a ying
al i udes o e alua e i lase poin s indeed all wi hin he bu e ed mask. The h ee anno a ed lase poin s p o ide
he a e age pai -wise dis ance dlp be ween lase poin s in pixel uni s.
To de ec he ed lase poin s in an image I, i s a linea combina ion o i s colo channels is used o gene a e
a lase signal image
I(LS)
:
Equa ion1 was de i ed om he obse a ion ha since he pixels a ound he lase poin s we e b igh ed
in colo (high alues in he ed colo channel), i ollows logically ha he blue and g een colo channels had
low alues in he same egion o he image. The e o e, sub ac ing a linea combina ion o blue and g een colo
channels om he ed colo channel would esul in an in e media e image, which has e y high alues a ound
he lase poin s and e y low alues elsewhe e.
(1)
I(LS)=I(R)−
0.2
(I(B)+I(G))
Table 1. O e iew o he OFOS deploymen s in he Cla ion Clippe on Zone du ing c uise SO268.
S a ion Con ac a ea Dep h (m)
App ox. ack leng h
(km)
App ox. bo om ime
(h)
Numbe o pic u es
aken a he sea loo
SO268-1_21-1_
OFOS02 Ge man 4538 9.2 11.0 3921
SO268-1_30-1_
OFOS03 Ge man 4070 10.7 12.0 4981
SO268-2_100-1_
OFOS05
Ge man D edge
(be o e) 4247 5.0 8.6 2749
SO268-2_117-1_
OFOS06 Ge man 4109 7.7 8.5 2956
SO268-2_126-1_
OFOS07 Ge man D edge (a e ) 4117 8.8 10.0 3492
SO268-2_160-1_
OFOS11 Ge man D edge (a e ) 4115 11.0 10.0 3526
SO268-2_164-1_
OFOS12 Ge man D edge (a e ) 4118 5.0 9.0 3075
SO268-2_177-1_
OFOS13 Ge man 4127 7.3 9.5 3414
SO268-1_63-1_
OFOS04 Belgian 4478 10.5 12.5 4532
SO268-2_128-1_
OFOS08 Belgian 4486 5.5 8.0 2749
SO268-2_147-1_
OFOS09 Belgian 4519 5.8 8.0 2981
SO268-2_153-1_
OFOS10 Belgian 4522 6.0 8.0 2302
1
Vol.:(0123456789)
Scien i 㘶c Repo s | (2023) 13:8350 | h ps://doi.o g/10.1038/s41598-023-35518-5
www.na u e.com/scien i ic epo s
An au oma ed image‑based
wo k 㘶ow o de ec ing
megaben hic auna in op ical
images wi h examples
om he Cla ion–Clippe on Zone
Benson Mbani
1*, Valen in Buck
1 & Jens G eine
1,2
Recen ad ances in op ical unde wa e imaging echnologies enable he acquisi ion o huge numbe s
o high‑ esolu ion sea 㘶oo images du ing scien i 㘶c expedi ions. While hese images con ain aluable
in o ma ion o non‑in asi e moni o ing o megaben hic auna, 㘶o a and he ma ine ecosys em,
adi ional labo ‑in ensi e manual app oaches o analyzing hem a e nei he easible no scalable.
The e o e, machine lea ning has been p oposed as a solu ion, bu aining he espec i e models s ill
equi es subs an ial manual anno a ion. He e, we p esen an au oma ed image‑based wo k 㘶ow o
Megaben hic Fauna De ec ion wi h Fas e R‑CNN (FaunD‑Fas ). The wo k 㘶ow signi 㘶can ly educes
he equi ed anno a ion e 㘶o by au oma ing he de ec ion o anomalous supe pixels, which a e
egions in unde wa e images ha ha e unusual p ope ies ela i e o he backg ound sea 㘶oo . The
bounding box coo dina es o he de ec ed anomalous supe pixels a e p oposed as a se o weak
anno a ions, which a e hen assigned seman ic mo pho ype labels and used o ain a Fas e R‑CNN
objec de ec ion model. We applied his wo k 㘶ow o example unde wa e images eco ded du ing
c uise SO268 o he Ge man and Belgian con ac a eas o Manganese‑nodule explo a ion, wi hin
he Cla ion–Clippe on Zone (CCZ). A pe o mance assessmen o ou FaunD‑Fas model showed
a mean a e age p ecision o 78.1% a an in e sec ion‑o e ‑union h eshold o 0.5, which is on a
pa wi h compe ing models ha use cos ly‑ o‑acqui e anno a ions. In mo e de ail, he analysis o
he mega auna de ec ion esul s e ealed ha ophiu oids and xenophyopho es we e among he
mos abundan mo pho ypes, accoun ing o 62% o all he de ec ions wi hin he su eyed a ea.
In es iga ing he egional di 㘶e ences be ween he wo con ac a eas u he e ealed ha bo h
mega aunal abundance and di e si y was highe in he shallowe Ge man a ea, which migh be
explainable by he highe ood a ailabili y in o m o sinking o ganic ma e ial ha dec eases om
eas ‑ o‑wes ac oss he CCZ. Since hese 㘶ndings a e consis en wi h s udies based on con en ional
image‑based me hods, we conclude ha ou au oma ed wo k 㘶ow signi 㘶can ly educes he equi ed
human e 㘶o , while s ill p o iding accu a e es ima es o mega aunal abundance and hei spa ial
dis ibu ion. The wo k 㘶ow is hus use ul o a quick bu objec i e gene a ion o baseline in o ma ion o
enable moni o ing o emo e ben hic ecosys ems.
Mode n digi al unde wa e imaging pla o ms such as he Ocean Floo Obse a ion Sys ems (OFOS)1, o Au o-
ma ed Unde wa e Vehicles (AUVs)2 a e inc easingly used o he explo a ion and moni o ing o ma ine seabed
ecosys ems by esea che s, he mili a y, as well as o he s akeholde s in he p i a e sec o 2. This is because hese
pla o ms o e a o dabili y, ease o deploymen , and he abili y o epea able sea loo sampling ac oss a ying
scales wi h high empo al and spa ial esolu ion3. As a esul o he ecen echnological de elopmen s in bo h
ha dwa e and so wa e, hese imaging pla o ms a e nowadays i ed wi h la ge memo y s o age capabili ies, as
well as high- esolu ion pho o and ideo came a senso s4,5. Consequen ly, came a deploymen s du ing scien i ic
expedi ions now gene a e huge olumes o high- esolu ion images o he sea loo 6. These images ca y a lo o
OPEN
1DeepSea Moni o ing G oup, GEOMAR Helmhol z Cen e o Ocean Resea ch Kiel, Wischho s aße 1-3, 24148 Kiel,
Ge many. 2Ins i u e o Geosciences, Kiel Uni e si y, Ludewig-Meyn-S . 10-12, 24118 Kiel, Ge many. *email:
bmbani@geoma .de
2
Vol:.(1234567890)
Scien i 㘶c Repo s | (2023) 13:8350 | h ps://doi.o g/10.1038/s41598-023-35518-5
www.na u e.com/scien i ic epo s/
aluable in o ma ion and insigh s in o deep sea ecosys em, such as he cha ac e is ics o sea loo subs a e7, as
well as he megaben hic auna ha inhabi s hese ecosys ems8.
Howe e , he lack o au oma ed echniques o analyzing and in e p e ing hese huge olumes o image da a-
se s limi s bo h he quali y and quan i y o in o ma ion ha can be de i ed om hem e.g., by ma ine scien is s
ocusing on deep sea geological and ecosys em moni o ing9. Fu he mo e, he cu en manual app oaches ha
in ol e he inspec ion and in e p e a ion o each image by a human analys a e no longe easible in his huge
da a egime, because manual anno a ion is expensi e, subjec i e and hus p one o human bias10. Despi e hese
challenges, unde wa e imaging has shown ema kable capabili y o documen ing new disco e ies in he deep
ocean, using bo h colo images and ideos. In pa icula , he use o unde wa e imaging in scien i ic publica ions
om domains such as ma ine ecological moni o ing, animal beha io obse a ion, and ime-lapse imaging o
empo al s udies, is es ima ed o ha e inc eased wo- old5. This p e e ence o ma ine imaging o e adi ional
sampling is as a esul o he abili y o pho og aphs o ep esen mo e axa, and also because he spa ial ex en
o he su eyed a ea can be de e mined accu a ely11. The e o e, au oma ed wo k lows a e needed o analyze
he acqui ed unde wa e images o suppo hese domain-speci ic applica ions. Depending on he applica ion,
hese au oma ed wo k lows can in ol e asks such as seman ic/ins ance segmen a ion, image classi ica ion, as
well as objec de ec ion.
Machine lea ning echniques ha e demons a ed he po en ial o au oma e bo h unde wa e image classi ica-
ion and objec de ec ion asks12. While image classi ica ion in ol es assigning a single class label o desc ibe he
con en o an en i e image scene (e.g. a habi a class), objec de ec ion goes u he o include he iden i ica ion
and localiza ion o indi idual ins ances o objec s isible in he image, ypically by d awing bounding boxes
a ound hem13. This makes objec de ec ion models pa icula ly use ul o ma ine scien is s who aim a iden i y-
ing, measu ing and coun ing unde wa e objec s e.g. o es ima e hei densi y and abundance14. While mode n
objec de ec ion models such as Fas e R-CNN15 can be ained o de ec objec s in images wi h ela i ely high
accu acy16, hey equi e a lo o manually anno a ed bounding box coo dina es along wi h hei co esponding
class labels, which is e y expensi e and edious o ob ain17. E en when expe anno a o s a e a ailable, he
selec ion o example images con aining mega auna o be p esen ed o he anno a o s can be e y challenging;
his is mo e p onounced in unde wa e image da ase s o deep seabed a eas because he equency and di e si y
o mega auna is e y low a g ea e dep hs, which implies ha only a small p opo ion o he unde wa e image
da ase con ain isible mega auna18. In OFOS/AUV deploymen s whe e ens o hund eds o housands o images
ha e been eco ded, he ask o selec ing example images wi h isible mega auna does pose a se ious challenge.
A p oposed wo k low o au oma ed de ec ion o megaben hic auna should he e o e inco po a e (semi)
au oma ed ways o educing and/o complemen ing he e o o human anno a o s e.g. by e icien ly expedi ing
he gene a ion o anno a ions om he op ical unde wa e images19,20. This au oma ion should acili a e bo h he
selec ion o example images wi h isible mega auna o be p esen ed o he anno a o s, as well as he gene a ion o
a se o weak anno a ions o be e ined la e . In his con ex , weak anno a ions a e imp ecise o noisy anno a ions
ha can be ob ained cheaply using unsupe ised app oaches21. An example o a se o weak anno a ions would
be bounding box coo dina es ha only pa ly co e he body o an ophiu oid (e.g., i s cen al disk) while lea ing
ou i s a ms. When a ailable, hese weak anno a ions g ea ly educe he e o equi ed om expe anno a o s,
since hei asks a e essen ially educed o: (a) e ining he p o ided bounding box coo dina es o p ecisely co e
he en i e megaben hic auna; (b) anno a ing addi ional megaben hic auna ha a e no pa o he p o ided
weak anno a ions; and (c) assigning he co ec mo pho ype class labels17. One compu a ionally cheap way o
gene a ing hese weak anno a ions is h ough he analysis o image supe pixels22.
Supe pixels a e pa i ions o an image whe e each pa i ion comp ises a g oup o pixels wi h simila pe cep ual
cha ac e is ics23. In unde wa e images eco ded om a ela i ely homogenous seabed subs a e e.g. sandy o
muddy bo oms in he deep sea, hese supe pixels gene ally co espond o he objec s occu ing on he sea loo ,
such as megaben hic auna, ocks o ma ine li e 24. Since he equency o megaben hic auna on he deep sea-
loo is e y low compa ed o backg ound objec s such as he so sedimen o ock deb is18, hose supe pixels
ha co espond o megaben hic auna can be conside ed anomalous. This is because hei isual p ope ies a e
clea ly di e en o he backg ound sea loo 25. Howe e , in o de o au oma ically dis inguish be ween no mal
and anomalous supe pixels, hei isual p ope ies mus i s be ex ac ed and encoded in o ea u e ec o s.
Al hough his can be achie ed by manually iden i ying he dis inguishing cha ac e is ics o he supe pixels
(e.g. colo and ex u e), his p ocess equi es signi ican amoun o domain expe ise and expe ience o be done
co ec ly6. An al e na i e app oach is o au oma ically lea n hese p ope ies di ec ly om he supe pixels e.g.,
by using con olu ional a ia ional au oencode s o ea u e ex ac ion26. Anomaly de ec ion algo i hms such as
iFo es 27 can hen be applied o hese ea u es, so ha anomalous supe pixels can be de ec ed and p esen ed o
expe anno a o s as weak anno a ions o e inemen , labeling, and subsequen aining o an objec de ec ion
model e.g., Fas e R-CNN15.
Pas s udies ha e p oposed a ious app oaches o sea loo subs a e classi ica ion28,29, and in pa icula o
unde wa e objec de ec ion. T adi ional image p ocessing echniques ha e been used o es ima e he co e age
o seag ass meadows in C oa ia h ough classi ica ion o i egula image segmen s30, as well as in he Palma bay
using egula squa e image iles31. In he same di ec ion, a saliency-based wo k low was implemen ed o app oxi-
ma e backg ound egions o he image o de ec unde wa e ‘ o eg ound’ objec s32, whe eas con as s e ching
and adap i e h esholding has been used o segmen and subsequen ly de ec unde wa e objec s33. Fu he ,
a combina ion o Laplacian il e ing, his og am equaliza ion and blob de ec ion has also been used o de ec
unde wa e objec s34. Rega ding he de ec ion o objec s and human a i ac s on he sea loo , a egion-based
app oach was used o de ec ma ine li e in G eek wa e s24, whe eas geome ic easoning was employed o he
de ec ion o pipelines on he seabed35. In ano he s udy36, unde wa e obo s we e used o pe o m colo es o-
a ion in eal ime in o de o imp o e accu acy when de ec ing and acking mobile objec s, whe eas empla e
ma ching was used o de ec and ack objec s om images eco ded using an unde wa e obo pla o m37. By
3
Vol.:(0123456789)
Scien i 㘶c Repo s | (2023) 13:8350 | h ps://doi.o g/10.1038/s41598-023-35518-5
www.na u e.com/scien i ic epo s/
modeling he p opaga ion o ligh h ough he wa e column, a wo k low was implemen ed o de ec unde wa e
objec s using monocula ision38, while ano he one was p oposed o de ec unde wa e objec s by le e aging
collima ed egions o a i icial ligh ing39. Mos ecen s udies employ deep lea ning app oaches: an a chi ec u e
was p oposed o de ec ing objec s in complex unde wa e imaging en i onmen s based on ea u e enhancemen s
and ancho e inemen 40, whe eas an augmen a ion s a egy was used o simula e e.g. o e laps and occlusions o
imp o e unde wa e objec de ec ion accu acy41. Simila ly, an a chi ec u e was p oposed o de ec unde wa e
objec s by accoun ing o unde wa e image deg ada ion h ough he join lea ning o colo con e sion and
objec de ec ion42. Finally, a a ia ional au oencode a chi ec u e was used o dis inguish salien egions om
he backg ound based on econs uc ion esiduals25.
In his s udy, we p opose a h ee-s age wo k low o au oma ically de ec ing megaben hic auna om op ical
unde wa e images; examples o a ge megaben hic auna classes (mo pho ypes) o his s udy a e shown in
Fig.1A, whe eas he p oposed wo k low is concep ualized schema ically in Fig.1B.
The i s s age in ol es gene a ing supe pixels om a small subse comp ising e.g., 500 images pe di e/cam-
e a ow, which a e andomly sampled o educe compu a ional cos in his s age. A a ia ional au oencode is
Figu e1. O e iew o ou op ical image-based megaben hic auna de ec ion amewo k. (A) Examples o
a ge mo pho ypes, including li e , ha we e de ec ed on he sea loo . (B) Schema ic diag am o ou h ee-
s ep wo k low: The i s s ep (au oma ically) gene a es supe pixels om a small subse o sampled images,
and (au oma ically) ex ac s hei ea u es o aining an anomaly de ec ion model. The second s ep de ec s
anomalous supe pixels (au oma ically) om a la ge subse o images, and (semi-au oma ically) p oposes hem
as weak anno a ions eady o be pos -p ocessed and assigned seman ic mo pho ype labels (manually). The inal
s ep uses he seman ic anno a ions o (au oma ically) ain a Fas e R-CNN objec de ec ion model, which hen
de ec s ins ances o ben hic mega auna isible in he en i e unde wa e image da ase (au oma ically), allowing
o he es ima ion o mega aunal abundance, di e si y and spa ial dis ibu ion (manually).
4
Vol:.(1234567890)
Scien i 㘶c Repo s | (2023) 13:8350 | h ps://doi.o g/10.1038/s41598-023-35518-5
www.na u e.com/scien i ic epo s/
hen applied o hese supe pixels o ex ac ea u e ec o s, which a e used o ain an iFo es anomaly de ec ion
model. The second s age applies he ained iFo es model di e-by-di e o de ec anomalous supe pixels om a
much la ge subse o unde wa e images e.g., comp ising six ou o wel e di es. A bina y classi ie is used o
pos -p ocessing he anomalous de ec ions o emo e alse posi i es. The bounding boxes o he uly anoma-
lous supe pixels a e hen p esen ed as a se o weak anno a ions o an expe anno a o , who assigns seman ic
mo pho ype labels o hem. The inal s age uses he seman ic anno a ions o ain and e alua e a Fas e R-CNN
objec de ec ion model, which is subsequen ly used o de ec and classi y mega auna isible in all images om
all di es. These geo e e enced de ec ions a e inally used o es ima e abundance, di e si y and spa ial dis ibu ion
o megaben hic auna wi hin he wo king a ea.
Ou app oach signi ican ly educes he equi ed human anno a ion e o , since he use inpu is only equi ed
o pos -p ocess he au oma ically gene a ed weak anno a ions, and assign hem seman ic labels. Fu he mo e,
we ha e also open sou ced he py hon sc ip s implemen ing each componen o he abo e-desc ibed wo k low,
along wi h de ailed documen a ion o guide use s o ge s a ed using and/o ex ending ou wo k low. Thus, ou
app oach o e s a con enien unde wa e image anno a ion solu ion o he ma ine imaging communi y, allowing
hem o quickly gene a e accu a e baseline in o ma ion ha allows o e icien and epea able cha ac e iza ion
o ecological and spa ial dis ibu ion o emo e ma ine ben hic communi ies, including hei habi a s, a a ying
spa io- empo al scales.
Resul s
Visualiza ion o supe pixel sepa a ion. This sec ion p o ides p ojec ions o bo h no mal and anoma-
lous supe pixels on o a wo-dimensional ea u e space o isualiza ion pu poses. These p ojec ions a e ob ained
by applying P incipal Componen s Analysis (PCA) on o he da a ma ix o ea u e ec o s ex ac ed om he
supe pixels. A g id iew o uly anomalous supe pixels is also p o ided.
Supe pixels o aining he iFo es anomaly de ec o . Figu e2 shows he ea u e space ep esen a ion o supe -
pixels used o ain he iFo es model. The igu e clea ly shows ha supe pixels ep esen ing he backg ound
sea loo a e densely dis ibu ed a ound he cen e o he ea u e space since hey a e isually simila , while
hose wi h unusual isual cha ac e is ics a e dis ibu ed a he away owa ds he pe iphe y o he ea u e space.
The e o e, he backg ound sea loo supe pixels a e ob iously he majo i y, and we e conside ed he ‘no mal’ in
his s udy.
De ec ed anomalous supe pixels. Figu e3 shows he ea u e space ep esen a ion o he anomalous supe pixels
ha we e de ec ed om di e 126. While some o he de ec ed anomalies a e alse posi i es e.g., he ed lase
Figu e2. Fea u e space p ojec ion o supe pixels whose ea u es we e used o ain he anomaly de ec ion
model. Those ep esen ing he backg ound sea loo a e densely dis ibu ed a ound he o igin o he ea u e
space, whe eas ew anomalous supe pixels a e spa sely dis ibu ed u he away owa ds he pe iphe y o he
ea u e space.
5
Vol.:(0123456789)
Scien i 㘶c Repo s | (2023) 13:8350 | h ps://doi.o g/10.1038/s41598-023-35518-5
www.na u e.com/scien i ic epo s/
poin s and unusually da k objec s on he seabed, he es o he de ec ed anomalies indeed ep esen in e es ing
objec s e.g., megaben hic auna, o o he unusual objec s wo h in es iga ing.
Weak anno a ions ( uly anomalous supe pixels). As men ioned abo e, some o he de ec ed anomalous supe -
pixels a e alse posi i es ha do no ep esen megaben hic auna. Thus, i was necessa y o emo e hese alse
posi i es, and e ain only he uly anomalous supe pixels du ing u he p ocessing. Below, we p o ide he
esul s o wo pos -p ocessing s a egies ha we a emp ed: se ing a h eshold on he anomaly sco e; and ain-
ing a supe ised bina y classi ie .
Figu e4A shows he esul s o pos p ocessing ob ained by se ing a 75 h pe cen ile h eshold on he anomaly
sco es assigned o he anomalous supe pixels; supe pixels wi h anomaly sco es g ea e han he se h eshold
we e ma ked as uly anomalous. While he supe pixels a e isually anomalous in some way, some o hem s ill
ep esen objec s ha a e no o in e es in his s udy e.g., he ed lase poin s, and whi e spo s su ounded by
black pixels. Because o his, we concluded ha h esholding based on anomalous sco es alone was no su icien
o dis inguish uly anomalous supe pixels om alse posi i es. The e was also no ob ious way o de e mining
he sui able anomaly sco e h eshold.
Figu e4B shows he uly anomalous supe pixels ha we e ob ained by using ou supe ised bina y classi ie .
Unlike he h esholding app oach, he bina y classi ie co ec ly iden i ied he se o uly anomalous supe pixels.
Bounding box coo dina es o hese uly anomalous supe pixels we e hen p oposed as a se o weak anno a ions.
T aining and e alua ing FaunD‑Fas model. The weak anno a ions s ill lack seman ic mo pho ype
labels, and a e he e o e no di ec ly usable. In his sec ion, we p o ide he esul s o he seman ic labeling
exe cise in ol ing an expe anno a o , as well as he esul s o he pe o mance e alua ion o he Fas e R-CNN
objec de ec ion model ha was ained using hese anno a ions.
Seman ic labeling o he weak anno a ions. A human expe manually inspec ed all he weak anno a ions and
assigned hem seman ic mo pho ype labels. The expe also anno a ed ins ances o mega auna ha we e isible
in he images, bu missing om he weak anno a ions. This seman ic labeling exe cise was epea ed wice (a e
shu ling he weak anno a ions) o educe biases e.g., due o human a igue.
Supplemen a y Figu eS1 shows a sc eensho o ou supe pixel anno a ion so wa e du ing an ac i e seman ic
labeling session. The le panel o he so wa e shows all uly anomalous supe pixels, whe eas he igh panel
displays hei bounding box ex en s o e laid on he espec i e pa en images. The bo om panel shows he
mo pho ypes ha we e conside ed in his s udy. These include: anemone, co al, ish, gas opod, holo hu ian,
ophiu oid, sea u chin, sh imp, sponge and xenophyopho e.
Figu e3. Fea u e space p ojec ion o he anomalous supe pixels de ec ed om images in di e 126. While
some alse posi i es such as ed lase s and da k pixels o he wa e column we e also de ec ed, he es o he
anomalous de ec ions ep esen po en ial ins ances o mega auna whose bounding boxes can be p oposed as a
se o weak anno a ions.
6
Vol:.(1234567890)
Scien i 㘶c Repo s | (2023) 13:8350 | h ps://doi.o g/10.1038/s41598-023-35518-5
www.na u e.com/scien i ic epo s/
Figu e5 shows he dis ibu ion o he anno a ed mo pho ypes. In e ms o p opo ions, he dominan mo -
pho ypes we e ophiu oids (31%), sponges (18%), xenophyopho es (17%) and anemones (11%). The o he mo -
pho ypes had occu ences o less han 10%. These a e he anno a ions we used o ain and e alua e ou FaunD-
Fas model; we ha e p o ided hese anno a ions as a cs ile in supplemen a y TableS1.
Pe o mance e alua ion o FaunD-Fas model. Ou FaunD-Fas model achie ed an a e age p ecision (AP.50)
sco e o 78.1% a an IoU h eshold o 0.5. The model pe o mance was highe when de ec ing la ge-sized objec s/
mega auna, as can be shown by he alues o (APla ge) and (ARla ge) me ic ca ego ies ha a e bo h g ea e o equal
o 70% (see Table1). On he o he hand, he model’s pe o mance was lowe when de ec ing small-sized objec s,
since bo h hei a e age p ecision (APsmall) and ecall (ARsmall) alues we e less han 20%.
When compa ed o compe ing s a e-o - he a models om he empi ical e alua ion inLü jense al43, hei
bes model (CM-X-101/Syn h-Blcd) pe o med be e han ou s wi h ega ds o he (AP.50:0.95) me ic ca ego y,
which is ob ained by a e aging he p ecision alues o e mul iple IoU h esholds. In con as , ou model pe -
o med be e han all he compa ed models wi h ega ds o he (AP.50), which is he p ecision a a single (abso-
lu e) IoU h eshold o 0.5. In addi ion, ou model also pe o med be e han he o he s wi h ega ds o he
Figu e4. G ids o image pa ches showing uly anomalous supe pixels ob ained by (A) Th esholding he
anomaly sco es, and (B) Bina y classi ie ained wi h examples o bo h ue and alse posi i es. Th esholding
p oduces undesi ed esul s e.g., he ed lase poin s and he da k pa ches om he wa e column. On he
o he hand, he bina y classi ie esul s in a se o uly anomalous supe pixels ha a e clea ly ins ances o
megaben hic auna. These we e p oposed as weak anno a ions.
Figu e5. Dis ibu ion o he anno a ed mo pho ypes a e expo ing om he anno a ion so wa e.
Ophiu oids, sponges and xenophyopho es we e among he dominan mo pho ypes in he anno a ed da ase .
7
Vol.:(0123456789)
Scien i 㘶c Repo s | (2023) 13:8350 | h ps://doi.o g/10.1038/s41598-023-35518-5
www.na u e.com/scien i ic epo s/
(APla ge) and (AR1) me ic ca ego ies; his implies ha ou model was good a de ec ing la ge-sized mega auna.
Howe e , all he compa ed models epo ed e y low p ecision and ecall sco es when de ec ing small-sized
objec s.
To show ou model’s pe o mance on he seman ic mo pho ype classes, we p esen he con usion ma ix in
Supplemen a y Figu eS2. The con usion ma ix shows ha majo i y o he mo pho ypes we e co ec ly local-
ized and iden i ied. In pa icula , xenophyopho es and ophiu oids con ibu ed owa ds he la ges p opo ion o
alse nega i es. This could be because he isual cha ac e is ics o some xenophyopho es and pa ially bu owed
ophiu oids a e simila o he seabed subs a e, which makes hem di icul o de ec . In addi ion o his, Fig.6
shows ha in ins ances cha ac e ized by associa ions among mo pho ypes e.g., be ween ophiu oids and sponges/
co als, he model made inco ec o low-con idence p edic ions.
Al hough none o he compa ed models (in Table1) was able o achie e he highes sco e ac oss all he me ic
ca ego ies, hese quan i a i e e alua ion esul s show ha o e all, he pe o mance o ou model was on a pa
wi h he bes pe o ming s a e-o - he a al e na i e(s), ye ou app oach equi ed less manual anno a ion e o .
Abundance, di e si y and spa ial dis ibu ion o he de ec ed megaben hic auna. Figu e7A
shows quali a i e examples o co ec ly iden i ied and localized mega auna as de ec ed by ou FaunD-Fas
model. In o al, 27,954 indi idual ins ances o megaben hic auna we e de ec ed om he en i e image da ase .
Fu he mo e, we es ima ed he mega aunal abundance wi hin Ge man a ea o be app oxima ely 0.247 ind. m−2
while in he Belgian a ea i was app oxima ely 0.200 ind. m−2.
Figu e7B shows he dis ibu ion o he de ec ed mo pho ypes. Ophiu oids and xenophyopho es we e he
mos dominan mo pho ypes accoun ing o 62% o all he de ec ions. O he species a e sponges (9.6%), sea
u chins (7.8%), gas opods (6.1%), anemones (5.7%), co als (4.0%) and holo hu ians (3.1%). The es such as
ish and sh imp ha e occu ences o less han 1%. Apa om ophiu oids which a e abundan in bo h con ac
a eas, he Ge man seabed is p edominan ly occupied by xenophyopho es (22.8%) and sponges (10%); he Bel-
gian con ac a ea was p edominan ly occupied by sea u chins (34.9%), anemones (14.5%) and sponges (10%).
Figu e7C shows ew examples o de ec ed mo pho ypes, while a able summa izing all he de ec ions is
p o ided in he Supplemen a y TableS2.
Table 1. Pe o mance compa ison ela i e o o he s a e-o - he a ben hic auna de ec ion models43. The
highes sco es pe me ic ca ego y a e indica ed in bold.
Model AP.50:.95 AP.50 APsmall APmedium APla ge AR1AR10 AR100 ARsmall ARmedium ARla ge
FaunD-Fas (Ou s) 46.5 78.1 12.7 42.0 69.7 50.0 52.0 52.4 16.2 50.0 73.2
CM-X-101/Baseline 41.7 68.2 25.3 29.3 54.7 21.6 51.6 55.2 25.4 45.1 70.8
CM-X-101/Syn h 48.8 71.0 27.4 39.1 62.8 24.7 58.8 64.2 27.9 57.3 77.1
CM-X-101/Syn h-Blcd 51.8 76.7 27.5 40.2 66.1 25.7 59.0 63.9 27.9 55.7 77.9
CM-X-101/T ad. Augm 48.8 75.0 26.9 38.6 58.5 23.0 55.3 58.9 27.2 50.1 72.6
CM-X-101/Fusion 51.7 74.1 27.1 42.1 65.1 24.9 57.6 61.6 27.5 52.2 77.6
CM-V-99/Syn h 47.9 72.0 27.9 37.0 62.8 23.6 56.6 61.9 28.3 52.6 77.1
CM-L-M/Syn h 27.3 48.6 19.1 19.0 40.0 18.3 39.1 43.7 20.0 34.4 59.5
M-X-101/Syn h 33.3 53.2 13.2 22.7 53.0 20.7 39.2 40.0 13.2 30.6 60.7
R-X-101/Syn h 47.8 70.7 27.9 37.1 62.2 24.2 56.6 61.9 28.4 53.8 76.7
Figu e6. Examples images showing co ec ly de ec ed ins ances o megaben hic auna, as well as ins ances o
bo h alse posi i es (FP) and alse nega i es (FN). Mo pho ypes whose isual cha ac e is ics is simila o he
sea loo subs a e (e.g. xenophyopho es and pa ially bu owed ophio oids) esul ed in a highe p opo ion o
alse nega i es. Also, inco ec de ec ion/localiza ion was obse ed in ins ances whe e mo pho ypes o med
associa ions wi h each o he e.g. be ween ophiu oids and sponges.
8
Vol:.(1234567890)
Scien i 㘶c Repo s | (2023) 13:8350 | h ps://doi.o g/10.1038/s41598-023-35518-5
www.na u e.com/scien i ic epo s/
The spa ial dis ibu ion o he de ec ed megaben hic auna is shown in Fig.8. The map shows ha he Ge -
man con ac a ea con ains a highe abundance o mega auna compa ed o he Belgian a ea (see de ails in he
“Discussion” sec ion below). Fo ease o isualiza ion, he de ec ions (poin s) we e spa ially clus e ed by i s
g idding hem in o squa e blocks o size 200m, and hen no malizing he absolu e coun o mega auna based
on he isual oo p in o each espec i e block (in squa e me e s). Thus, he symbology size is p opo ional o
he abundance o mega auna wi hin each spa ial clus e /block.
Discussion
The p oposed megaben hic auna de ec ion wo k low comp ised he gene a ion o weak anno a ions om supe -
pixels, seman ic mo pho ype labeling o he p oposed weak anno a ions, and inally he usage o hese anno a-
ions o ain ou FaunD-Fas model. Below, we discuss key aspec s o hese p oposed wo k low s eps, and p o ide
a mo e de ailed discussion o he spa ial dis ibu ion, densi y and di e si y o he de ec ed megaben hic auna.
We also sugges a ew ecommenda ions o u he esea ch.
The hype pa ame e se ings o he segmen a ion algo i hm con ol he geome ical p ope ies o he gene -
a ed supe pixels e.g., shape ( egula o i egula ), and size (la ge o small). Gi en ha he used sea loo images
comp ised backg ound seabed subs a e (Mn-nodules) and o he objec s o a ying shapes and sizes, we had o
manually de e mine he op imal alues o hype pa ame e s such as scale (pixel size) and wid h o he gaussian
il e ha smoo hs he image p io o segmen a ion. These pa ame e s mus be p ope ly uned i he wo k low is
applied o o he unde wa e image da ase s. I his is no done ho oughly, he gene a ed supe pixels may be o
low quali y hence nega i ely a ec ing he accu acy o downs eam analysis. In ou case, we obse ed ha a poo
choice o hese hype pa ame e s led o inaccu a e segmen a ion o ce ain mo pho ypes o in e es , especially
hose wi h ex ended a ms and spikes e.g., ophiu oids and sea u chins. In a ela ed p e ious s udy using a ish
da ase 44, he au ho s also emphasize ha segmen a ion hype pa ame e s mus be p ope ly op imized be o e
being applied o unde wa e images eco ded om challenging en i onmen s e.g. whe e bo h illumina ion condi-
ions and backg ound sea loo p ope ies a y wi hin and be ween da ase s. In addi ion o he hype pa ame e
se ings, we also had o selec a subse o images whose supe pixels would be used o ain he iFo es anomaly
de ec ion model; ou subse comp ised 500 andomly sampled images ha gene a ed 125,000 supe pixels. We
Figu e7. (A) Quali a i e examples o de ec ed ins ances o megaben hic auna (B) Dis ibu ion o
mo pho ypes ha we e de ec ed by ou FaunD-Fas model. This dis ibu ion is simila in shape o ha o
anno a ions (see Fig.5), excep he FaunD-Fas de ec ed a lo mo e ins ances o mega auna. (C) G id iew
showing mega auna examples g ouped by mo pho ypes in e e y ow o he g id; he mo pho ype label o each
ow ollows he same o de as in panel (B).
9
Vol.:(0123456789)
Scien i 㘶c Repo s | (2023) 13:8350 | h ps://doi.o g/10.1038/s41598-023-35518-5
www.na u e.com/scien i ic epo s/
chose his sample size because i i in o ou CPU memo y a ain ime (a la ge subse should be used i mo e
memo y and compu e is a ailable e.g., in HPCs). In any case, we obse ed ha his andom sampling app oach
po en ially esul ed in a mo e ep esen a i e subse compa ed o e.g., a manual sampling app oach. This is because
he huge olume o images could easily cause he analys o mis akenly choose a subse o images ha ep esen
mo e o less he same egion o he sea loo (consecu i e images we e eco ded e e y 10s, and a e s o ed on disk
in o de o hei acquisi ion ime), o hose which look ‘in e es ing’.
The majo i y o he supe pixels gene a ed o aining he anomaly de ec ion model ep esen ed backg ound
sea loo , compa ed o he ela i ely ew mega auna supe pixels (see Fig.2). This obse a ion was expec ed since
he abundance o megaben hic auna in he deep ocean is ypically e y low, due o he low o ganic ca bon
lux/li le ood a ailabili y in g ea wa e dep h18. Simila indings ha e been epo ed in s udies ha examined
he ela ionship be ween megaben hic auna communi ies and ba hyme ic g adien s e.g.45–48. As a esul , we
Figu e8. Map iew showing spa ial dis ibu ion o de ec ed megaben hic auna along came a deploymen
acks in bo h he Ge man and Belgian con ac a eas. The Ge man seabed con ained highe abundance o
mega auna, p obably because o a ailabili y o ood in o m o sinking o ganic ma e ial since i is on a e age
shallowe han he Belgian seabed. The map was gene a ed using he open sou ce QGIS so wa e 3.2 (h ps://
www. qgis. o g/).
10
Vol:.(1234567890)
Scien i 㘶c Repo s | (2023) 13:8350 | h ps://doi.o g/10.1038/s41598-023-35518-5
www.na u e.com/scien i ic epo s/
obse ed ha he de ec ion o anomalous supe pixels was ela i ely s aigh o wa d: hey a e isually a he
di e en om he backg ound, and a e hus dis ibu ed u he om he cen e o he ea u e space whe e he
majo i y o he backg ound supe pixels clus e ed (see Fig.3). The app oach o analyzing he isual p ope ies
o supe pixels has been employed in p e ious s udies aimed a iden i ying he bounda ies o in e es ing objec s
on sea loo images32,44, as well as on e es ial images25. In con as o hese h ee publica ions, ou app oach
is di e en because we do no make assump ions ega ding he egion o he image in which he o eg ound
objec s a e expec ed e.g., in he cen al po ion o he image. Ins ead, we assume ha ou objec s o in e es will
be loca ed anywhe e on he image, and hus ou ained iFo es model de ec s anomalous supe pixels based
pu ely on ea u es ex ac ed om he supe pixels. The de ec ed anomalies s ill had o be pos -p ocessed o
emo e alse posi i es, which occu ed because we in en ionally se iFo es ’s ‘con amina ion ac o ’ se ing o a
high alue (0.4); his caused i o lag as many anomalous supe pixels as possible (bo h ob ious and sub le). We
did his because he isual p ope ies o some mega auna o in e es such as xenophyopho es a e e y simila o
backg ound sea loo subs a e, ye we wan ed he iFo es model o de ec hem as well. In a p e ious s udy on
image-based mega auna communi y assessmen in he DISCOL a ea o he sou h Paci ic Ocean49, he au ho s
also poin ou he di icul y ha e en expe human anno a o s ace when i comes o dis inguishing ce ain
mo pho ypes om backg ound sea loo .
The bounding box coo dina es o he de ec ed anomalous supe pixels we e p oposed as a se o weak anno a-
ions o be inspec ed and labeled by an expe anno a o . This semi-au oma ed app oach signi ican ly educed
he human e o equi ed in gene a ing aining anno a ions, because i was no longe necessa y o he expe
anno a o o manually inspec a la ge numbe o images wi h he aim o iden i ying he ew ha con ain isible
auna, and hen ma k hese auna manually. The a ailable bounding box u he educed he wo k o he expe
anno a o s o jus e i ying and assigning seman ic mo pho ype labels. Since hese anno a ions we e gene a ed
om unsupe ised segmen a ion and anomaly de ec ion me hods, hey a e no su icien on hei own (in quan-
i y and quali y) o es ima e he abundance o mega auna on he seabed o he en i e da ase . They jus ep esen
aining examples o a s a e-o - he a objec de ec ion model, which can hen be applied o he en i e image
da ase . A e using he gene a ed anno a ions o ain ou FaunD-Fas model, we achie ed a good pe o mance
(78.1%) ha is on a pa wi h o he s a e-o - he a objec de ec ion models, which we e ained in p e ious
s udies43 using unde wa e image da ase compa able o ou s (see Table1). Gi en ha ou FaunD-Fas model
achie es his good pe o mance o a ac ion o he anno a ion e o implies ha i is scalable o o he applica-
ions in ol ing huge olumes o unde wa e image y. Ano he obse a ion is ha since ou model uses he wo-
s age Fas e R-CNN a chi ec u e ha p io i izes p edic ion accu acy o e speed15, he esul s o ou compa ison
wi h o he s a e-o - he a models (in Table1) shows ha ou app oach is sui able o deploymen on wo ks a-
ions wi h good p ocessing capabili y e.g. GPU and memo y esou ces. Fo as e de ec ion on edge de ices and
smalle compu e s, he compa ison implies ha a single s age de ec o is p obably mo e sui able; u u e esea ch
could explo e his u he . Also, none o he compa ed s a e-o - he-a models achie ed he highes sco e ac oss
all he me ics, which could be because each model a e ained o op imize a di e en loss unc ion50. Finally,
he compa ison e ealed ha he p edic ion accu acy o small-sized mega auna was consis en ly lowe han o
la ge-sized mega auna ac oss all he compa ed models. This could be caused by he con olu ions and pooling
laye s in he objec de ec ion a chi ec u e, which g adually educe he size (and esolu ion) o he image deepe
in o he ne wo k, making small sized objec s ha de o de ec 51. Fu u e esea ch could explo e backbone ne wo k
a chi ec u es ha imp o e he model’s de ec ion o small-sized objec s. Mo eo e , u u e esea ch could explo e
how o ex end he FaunD-Fas model o e-use image ea u es ex ac ed om ea lie s ages o he wo k low so
as o educe compu a ion cos , especially o eal ime mega auna de ec ion while a sea.
Based on he de ec ion esul s o ou ained FaunD-Fas model, we ound ha he megaben hic auna
abundance in ou wo king a ea was ela i ely low. This inding is consis en wi h p e ious s udies om he CCZ
ha also epo ed mega aunal abundances o less han one indi idual pe squa e me e 52–55. In e ms o egional
di e ences be ween he wo con ac a eas, a highe abundance was obse ed in he Ge man a ea (0.247 ind.
m−2) compa ed o he Belgian a ea (0.200 ind. m−2). Simila ly, he mega aunal di e si y was highe in he Ge man
a ea, wi h a Shannon di e si y index o 2.4 compa ed o he 1.7 in he Belgian a ea. Because he Ge man a ea
is loca ed app oxima ely 1050km eas o he Belgian a ea, he obse ed high di e si y and abundance could be
as a esul o he eas - o-wes educ ion in he pa icula e o ganic ca bon lux (POC), as has also been epo ed
in p e ious s udies56,57. Also, he di e ence in abundance be ween he wo con ac a eas could be explained
by a ailabili y o ood sou ce in he o m o sinking o ganic ma e ial h ough he wa e column; a ailabili y o
ood is highe in Ge man a ea because i is on a e age shallowe (−4121m) han he Belgian a ea (−4510m).
This ela ionship be ween ood a ailabili y and abundance o mega auna in he Paci ic has also been epo ed in
a p e ious s udy58. Conside ing he ela ionship be ween megaben hic auna abundance and he sea loo sub-
s a e classes om28, he Belgian a ea con ained mo e han 68% o de ec ed megaben hic auna occupying he
la ge-sized nodules, e en hough he seabed subs a e comp ised bo h la ge- and densely-dis ibu ed nodules.
A lowe p opo ion o megaben hic auna was obse ed in densely dis ibu ed manganese nodules, p obably
because his subs a e class does no allow enough space o so -sedimen dwelle s, as was also poin ed ou in
a p e ious s udy59. On he o he hand, analysis in he Ge man a ea e ealed ha 57% o megaben hic auna
occu ed in sea loo subs a es comp ising pa chy nodules. In addi ion o being he dominan sea loo class in
his a ea, pa chy nodules also p o ide a na u al balance be ween so and ha d subs a es, which would accom-
moda e bo h ha d and so sedimen dwelle s, as was also epo ed in a p e ious s udy59. In bo h con ac a eas,
we ound ha ophiu oids and xenophyopho es we e he mos abundan and di e se mo pho ypes, and ha hey
occu in associa ion wi h each o he , while occupying bo h ha d and so bo om subs a es. Simila conclusions
we e also d awn om p e ious s udies59–61.
!
Pee e iew s a us:!
This is a non-pee - e iewed p ep in submi ed o Ea hA Xi .
Au oma ed unde wa e image analysis e eals sedimen
pa e ns and mega auna dis ibu ion in he opical A lan ic
Benson Mbani1* and Jens G eine 1,2
1DeepSea Moni o ing G oup, GEOMAR Helmhol z Cen e o Ocean Resea ch Kiel,
Wischho s aße 1-3, 24148 Kiel, Ge many
2Ins i u e o Geosciences, Kiel Uni e si y, Ludewig-Meyn-S . 10-12, 24118 Kiel, Ge many.
* Co espondence:
Co esponding Au ho
bmbani@geoma .de
Abs ac
The deep sea en i onmen comp ises di e se o a, auna and habi a s, whose cha ac e isa ion is key
owa ds ou collec i e unde s anding o ocean heal h and esilience. Whe eas di ec sampling
allows o de ailed in es iga ion o he e ical a iabili y o seabed cha ac e is ics a small spa ial
scales, op ical imaging is sui able o high- esolu ion assessmen o he spa ial dis ibu ion o
habi a s and hei ben hic mega auna ac oss mul iple scales. These assessmen s a e ypically
acili a ed by scien ic expedi ions ha su ey ex ensi e seabed a eas using e.g. con inuous imaging
echniques, gene a ing huge olumes o high- esolu ion images o which manual inspec ion and
anno a ion is cos ly, non-scalable and he e o e in easible. T ans o ming hese e aby e-scale images
(and ideos) in o ac ionable insigh s equi es au oma ed wo kows ha expedi e bo h he
gene a ion o baseline in o ma ion, as well as downs eam spa ial-ecological analysis. He e, we
deployed wo A.I wo kows o au oma e he anno a ion o seabed subs a es and mega aunal axa
om s ill images, which we acqui ed du ing se en came a deploymen s along an 18° N Eas -Wes
sec ion in he opical A lan ic no h o Cabo Ve des. We manually inspec ed he au o gene a ed
anno a ions o quali y, and subsequen ly assigned hem seman ic labels. The ea e , we used
clus e ing, ea u e space isualisa ion and mul i a ia e s a is ical analysis echniques o classi y he
seaoo in o habi a s, es ima e mega aunal abundance and spa ial dis ibu ion pa e ns, as well as
en i onmen al d i e s ha inuence he iden ied pa e ns. Ou esul s show ha he seabed can
be pa i ioned in o se en clea ly dis inc clus e s, wi h each o hese clus e s showing isible
sub-pa i ions. In es iga ions e ealed a clea g adien in e ms o sedimen dis u bance due o
biogenic ac i i y, wi h images showing li le- o-no sedimen dis u bance g ouping oge he on one
hal o he ea u e space, whe eas hose images wi h isibly igo ous signs o sedimen ewo king
clus e ed on he o he hal . Ou esul s also show ha mega aunal abundance was on a e age 14
imes highe in he Eas e n egion o ou s udy a ea, which was app oxima ely 700 me es shallowe
and close o sho e han he Wes e n egion. This obse ed high abundance could be a ibu ed o
highe POC ux ha anspo s mo e o ganic ma e o he shallowe seabed, as well as due o
ela i ely wa me empe a u es ha enhance me abolic a es o ben hic auna. Ou esul s u he
e eal geog aphic ho spo s o mega auna in opog aphically complex ea u es such as he sides o a
subma ine canyon and he op o seamoun s. The complex opog aphy o hese ea u es in oduces
he e ogenei y ha c ea es di e se mic ohabi a s and unique niches ha mega auna exploi . Finally,
we obse ed ha while co- a ying dep h and longi ude a iables gene ally explained he sepa a ion
be ween he wo main mega aunal communi ies in ou Eas -Wes o ien ed wo king a ea,
ba hyme ic d i e s like slope and uggedness had a mo e p onounced inuence in he deepe
Wes e n egion (-3698m) compa ed o he shallowe Eas e n egion (-2477m deep). Collec i ely,
hese ndings demons a e ha he in eg a ion o A.I wo kows in o classical spa io-ecological
me hods does expedi e he ans o ma ion o la ge olumes o ma ine image da ase s in o
ac ionable insigh s, he eby signican ly con ibu ing o ou unde s anding, moni o ing and
sus ainable use o ocean esou ces.
In oduc ion
The deep sea comp ises a wide ange o ben hic habi a s, is home o di e se se s o o al and aunal
communi ies, and is he la ges biome on ea h1. Despi e his, he biodi e si y wi hin hese emo e
ecosys ems is s ill la gely unde -sampled2and/o pa chily documen ed3, e en a e accoun ing o
he inc eased equency o scien ic expedi ions o e he pas decades4. This is because o logis ical,
echnological and nancial challenges ha cons ain he o e all spa io- empo al ex en s ha can be
easonably in es iga ed 5. Besides, in-si u and/o isual cha ac e iza ion o o ganisms in he deep sea
can some imes be non- i ial, ei he because o ganisms in hese en i onmen s a e new o science, o
because hei dis ibu ion pa e ns (and ecosys em p ocesses) a e no ye p ope ly unde s ood 6.
Recen scien ic s udies ha e also p o ided conclusi e e idence showing a global decline in ma ine
biodi e si y as a esul o bo h na u al and an h opogenic ac o s e.g. o e shing, pollu ion, coas al
de elopmen , na u al clima e a iabili y, and long- e m geological p ocesses like sedimen a ion and
ec onic/hyd o he mal ac i i ies 7. To be e quan i y and add ess his biodi e si y decline, globally
coo dina ed eo s a e equi ed o no only inc ease he equency and spa ial ex en o ma ine
ecosys em su eys, bu also o expedi e he analysis o he acqui ed da ase s. These da ase s include
high esolu ion images and ideos collec ed using pla o ms such as Au onomous Unde wa e
Vehicles (AUVs), Remo ely Ope a ed Vehicles (ROVs), and owed Ocean Floo Obse a ion
Sys ems (OFOS) 8. While imaging senso s a ached on o hese pla o ms con enien ly allow o
non-in asi e su eying o deep-sea en i onmen s in high esolu ion, hey gene a e huge olumes o
image y o which manual in e p e a ion is un easible 9. As a esul , au oma ed wo kows based on
eme ging digi al echnologies a e equi ed o expedi e he p ocessing and anno a ion o images,
he eby p o iding comp ehensi e baseline in o ma ion on geological, sedimen ological and
biological p ope ies o ma ine ecosys ems10.
Mode n machine lea ning echniques ha e demons a ed he capaci y o a he quick ye accu a e
ex ac ion o seman ic in o ma ion om la ge sequences o image and ideo da ase s 11. In
pa icula , p e- ained compu e ision models based on con olu ional neu al ne wo ks a e
nowadays eadily a ailable o download om open-sou ce eposi o ies (e.g Tenso Flow Hub and
PyTo ch Hub), and can be di ec ly deployed as-is o accomplish common asks such as image
enhancemen , classica ion, objec de ec ion and dense pixel segmen a ion 12. Gi en ha mos o
hese p e- ained models we e o iginally ained o iden i y common objec s on e es ial images
using benchma k da ase s like ImageNe 13, he models equi e ne- uning using anno a ed
unde wa e images be o e hey can be use ul o applica ions such as ma ine habi a mapping and
biodi e si y assessmen 14. This equi emen poses signican bo lenecks in a leas wo
dimensions: Fi s , anno a ing images a e e e y scien ic expedi ion is cos ly, unscalable and
he e o e undesi able; Second, ma ine en i onmen s na u ally exhibi low densi y o mega auna
wi h inc easing dep h, which implies ha o ganisms will be isible on only a hand ul (ou o
possibly ens o housands) o acqui ed images ha a e ypically unknown ap io i 15. Add essing
hese challenges equi es au oma ed A.I-based seaoo classica ion and mega aunal de ec ion
wo kows ha no only wo k well in a specic wo king a ea, bu ha a e easily gene alizable o
o he ma ine ecosys ems 16. Such a gene alised app oach sa es human analys s he ouble o
anno a ing da ase s om sc a ch, allowing hem o concen a e on ening and assigning seman ic
mo phospecies labels only o au o-gene a ed anno a ions 9. The seman ic anno a ions can hen
o m he basis o downs eam assessmen o spa io-ecological dis ibu ion pa e ns o habi a s,
mega auna and en i onmen al d i e s.
Mega aunal species a e no dis ibu ed andomly in space 17. Ins ead, hey clus e oge he in o
bio ically-simila communi ies ha a e in u n s uc u ed by p ocesses and a iables such as
ba hyme ic g adien s, geomo phology, ood a ailabili y, chemical/physical bo om wa e
condi ions, as well as sedimen o ha dg ound p ope ies ela ed o se ling, hiding o b eeding 18.
Cha ac e iza ion o hese mega aunal pa e ns is ypically pe o med using mul i a ia e s a is ical
analysis echniques 19, which a e also applicable o his p esen s udy gi en ha ou image-de i ed
anno a ions comp ise abundances o mul iple axa. Be o e using hese s a is ical echniques o
assess he dis ibu ion o mega auna, howe e , i is necessa y o s accoun o he inconsis en
isual oo p in s among espec i e images due o hei a iable acquisi ion heigh s 16. This
inconsis ency can be esol ed by sys ema ically dening s anda dised sampling uni s (e.g equal-a ea
quad a s o xed-leng h linea ansec s), wi hin which mega auna coun s a e pooled and
no malised ela i e o he ac ual obse ed a ea 20. Collec i ely, hese sampling uni s encode bio ic
in o ma ion as abundances ha can simply be binned and plo ed on a cho ople h map o isualise
spa ial dis ibu ion o mega auna. Al e na i ely, o dina ion echniques such as non-me ic
mul idimensional scaling (nm-MDS) can be used o g aphically display in e - ela ionships among
he die en axa in ea u e space 19. Fu he mo e, an a bi a y numbe o ele an en i onmen al
a iables can also be supe imposed on he o dina ion plo , allowing o a mo e nuanced isual
assessmen o he (subse o ) abio ic ac o s ha inuence he die en clus e s o mega auna 21.
Finally, spa ial au oco ela ion analysis may also be used o e eal mega aunal ho spo s, coldspo s
and ou lie s 22.
Pas s udies ha e p oposed a ious wo kows and app oaches o semi-au oma ing he anno a ion
o unde wa e images. Supe ised app oaches ha e been used ex ensi ely o asks such as
image-based seaoo classica ion because hey a e capable o gene a ing accu a e anno a ions (in
in e ence mode) whene e sucien numbe o labelled examples a e a ailable o aining 23. To
acili a e apid inno a ion, expe imen a ion, ep oducibili y and e alua ion o supe ised models,
he e ha e been s udies aimed a cu a ing s anda dised (labelled) benchma k da ase s om bo h
eal 24 and simula ed ma ine en i onmen s 25. Whe eas classical machine lea ning echniques such as
andom o es s 26 and suppo ec o machines 27 we e p edominan ly inco po a ed in ma ine
image analysis wo kows in he pas decade, ecen s udies almos exclusi ely use con olu ion
neu al ne wo ks 28. In pa icula , models such as YOLO 29, Re inaNe 30 and Fas e R-CNN 31 a e
now widely used o de ec ing, localising and classi ying o a and auna om images and ideos
a e aining wi h jus hund eds o aining examples pe class 11. The e is also e idence ha hese
deep lea ning models a e compu a ionally esou ce-in ensi e only du ing model aining, o he wise
he models a e ema kably ecien when making p edic ions in in e ence mode 32. Unsupe ised
app oaches such as empla e ma ching 33 and supe pixel-based segmen a ion ha e also been used in
p e ious s udies 34, ypically as an ini ial p elimina y s ep e.g. o cheaply gene a e weak anno a ions
35, o o quickly so images based on na u al g oupings 36. Some s udies s ill ely exclusi ely on
human wo k o ce o exhaus i ely anno a e hei da ase s, which is accu a e (and a guably he gold
s anda d) bu also e y cos ly and non-scalable 37. Rega dless o he chosen anno a ion s a egy, he
gene a ed anno a ions a e no mally used as inpu s o downs eam spa io-ecological wo kows ha
ely on e.g mul i a ia e s a is ics and measu es o spa ial au oco ela ion o cha ac e ise
abundances, di e si y, and spa ial dis ibu ion pa e ns o mega auna 19 17 21
He e, we in es iga ed seaoo habi a s and ben hic mega aunal dis ibu ion pa e ns in he opical
No h A lan ic using he concep ual wo kow in Figu e 1. Specically, we ne- uned A.I
wo kows ha we p e iously de eloped o classi ying seaoo habi a s 9and de ec ing ben hic
mega auna 23 in he Cla ion-Clippe on Zone. We used he ne- uned models o expedi e he
anno a ion o a new da ase comp ising seaoo images om he opical No h A lan ic. Gi en
ha he A.I wo kows we e o iginally used o ben hic assessmen s in he Pacic, one b oad
objec i e o his s udy was o in es iga e he gene alizabili y o he wo wo kows when p esen ed
wi h da ase om a comple ely die en a ea and geological se ing. Specic objec i es we e: (1) o
e eal sub le a iabili y in seaoo habi a classes using unsupe ised machine lea ning echniques;
(2) o semi-au oma e he de ec ion, localisa ion and classica ion o megaben hic axa om
sequences o high- esolu ion images; (3) o cha ac e ise he spa ial dis ibu ion pa e ns o he
anno a ed mega auna; (4) o es ima e mega aunal abundance, di e si y and communi y
composi ion; and nally, (5) o assess he inuence o en i onmen al d i e s on he obse ed
dis ibu ion pa e ns.
Figu e 1: Flow diag am showing he in e connec ed componen s o ou p oposed wo kow ha
comp ises h ee key s eps: Fi s , we enhance he isibili y o images be o e deploying wo A.I
wo kows o classi y seaoo images in o habi a classes, and also o de ec mega aunal axa;
Second, we inspec and assign seman ic axa labels o he au o-gene a ed weak anno a ions,
be o e con e ing he absolu e axa coun s in o abundances ela i e o ac ual obse ed a ea
wi hin ou p edened xed-size sampling uni s; Finally, we use he abundances o cha ac e ise
spa ial dis ibu ion pa e ns o mega auna, and also o g aphically display in e ela ionships
among bio ic and abio ic a iables in o dina ion ea u e space.
S udy A ea
Ou wo king a ea osho e Mau i ania and No h o Cape Ve des (Figu e 2) ollowed an Eas -Wes
o ien a ion, wi h a o al o se en came a deploymen s a ions dis ibu ed be ween he Eas e n
egion (comp ising di es 131, 144, 145) and Wes e n egion (di es 19, 32, 28, 78).
The Eas e n egion was shallowe wi h dep hs anging be ween -2470 m up o -2970 m. This
egion was cha ac e ised by opog aphically complex ea u es such as a seamoun (in di e 145), as
well as he subma ine canyon a -2920 me es wa e dep h (in di e 144). The canyon exhibi s s eep
nea - e ical 20-me e-high walls wi h a c oss sec ion ha is app oxima ely 500 me es wide,
ma king a isibly dis inc na ow passage on he seabed. CTD p oles u he show ha he wa e
masses in he Eas e n egion a e ela i ely wa me , wi h a e age empe a u es o (2.85°C ± 0.12).
In con as , he Wes e n egion was deepe and ela i ely colde , wi h dep h anges o be ween
-3128m and -3693 m, and a e age empe a u es o (2.50°C ± 0.08). The e was also a seamoun in
his egion ha ose app oxima ely 200 m high om he seabed (in di e 32), as well as a pai o
adjacen 40-me e high locally ele a ed abyssal hills (in di e 28). Fo u he de ails on he physical
and wa e mass p ope ies o espec i e di es, please e e o he CTD p oles in Supplemen a y
Figu e S1.
Figu e 2: Map showing he OFOS (came a) deploymen acks du ing c uise M182 o he
opical No h A lan ic, which we conduc ed on boa d RV Me eo be ween May - July 2022.
No e ha in his map we only show came a deploymen s om deep sea en i onmen s (> 2,000
me es wa e dep h). Also no ice ha di e 145 is sho e han he o he di es because he came a
mal unc ioned a e jus 1 hou 13 minu es o bo om ime.
Image da ase
We su eyed he wo king a ea be ween May 31s and July 10 h 2022 on boa d RV METEOR
du ing expedi ion M182 (G eine e al., 2024; link o c uise epo o la e ). The aim o he c uise
was o s udy he inuence o mesoscale eddies on (a) biogeochemical p ocesses in he Eas e n
bounda y upwelling sys ems, and (b) modula ion o o ganic ma e anspo om he su ace
wa e s down o he seaoo 6. The sampling campaign in ol ed he deploymen o se e al gea s and
sys ems such as he ex ended Ocean Floo Obse a ion Sys ems (XOFOS), CTDs, Mul iNe s,
biogeochemical lande s, AUVs, and ship-based mul ibeam ba hyme y.
Fo his pa icula s udy, we used s ill images collec ed by he XOFOS, which is an imaging
pla o m comp ising a opside uni on he ship ( o powe , da a connec ion and li e ideo eed), as
well as a subsea uni ha is lowe ed in o he wa e column by a winch sys em o su ey he seaoo
(up o 6000 me es deep). The subsea uni comp ises a hea y me al ame ha houses o wa d- and
downwa d-looking 24-megapixel digi al Ocean Imaging Sys ems came a (DSC 24000). The
XOFOS eco ds Images au oma ically a a cons an equency ha is se be o e deploymen , as well
as h ough ho key unc ionali y o eco ding adhoc images o andom e en s o in e es . In
addi ion o he came a, he XOFOS is also ed wi h downwa d acing LED ligh s/ashe s and a
USBL posi ioning sys em ha acks he pla o m posi ion du ing image acquisi ion. Addi ional
senso s such as ADCPs, CTDs and o he logge s may also be a ached o he XOFOS, allowing o
a s aigh o wa d in eg a ion o auxilia y da ase s and images based on synch onised imes amps.
Based on he abo e se up, we ob ained 8838 s ill images by pho og aphing he seabed a cons an
equencies o 0.07 Hz (in di es 28, 32), 0.10 Hz (in di es 78, 131, 144, 145) and 0.2 Hz (in di e
19). The se en XOFOS di es co e ed a o al ack leng h o 22.7 kilome es, which ep esen s a
isual oo p in o app oxima ely 73,616 m2on he seaoo . We es ima ed his isual oo p in
based on he xed opening angles o he came a (48° ho izon al, 33° e ical) and he acquisi ion
heigh s o espec i e images abo e he seaoo . Table 1 below p o ides an o e iew o ou came a
deploymen s, while he c uise epo con ains u he echnical de ails ega ding he image
acquisi ion se up (G eine e al., xxxx).
Figu e 4: Va iabili y in numbe o images, XOFOS speed, and ac ual obse ed a ea wi hin
xed-size (100-me e-long) sampling uni s along espec i e su ey ansec s. (A) shows an
ob ious in e se ela ionship be ween he speed o he XOFOS and he numbe o acqui ed
images, whe eas in (B) he ela ionship be ween numbe o images and obse ed isual oo p in
is no ob ious, especially in opog aphically complex e ains like seamoun s. No e ha he
ela i ely high numbe o images a he s a o some ansec s is caused by he ini ial s abilisa ion
phase, whe e he deployed XOFOS s expe iences wis s and u ns in mo e o less he same
loca ion be o e e en ually main aining a linea ansec .
Assessing he spa ial dis ibu ion o mega auna
Cha ac e ising he spa ial dis ibu ion o megaben hic auna allows us o p o ide geog aphic
con ex o he image-de i ed abundances. He e, we used he cen oid coo dina es o each sampling
uni o plo hei loca ions in map iew. We applied quan ile classica ion o bin abundances in o
eigh dis inc classes ha we used o colou -code he cho ople h maps (Figu e 13). This isual
ep esen a ion allowed o a s aigh o wa d in e p e a ion o he a iabili y in mega aunal
abundances ela i e o he backg ound ba hyme y ha we plo ed as a basemap. In addi ion o he
plana map iew, we also plo ed he abundances along ele a ion p oles o each ansec , o
in es iga e a iabili y a local heigh s.
To complemen he quali a i e cho ople h mapping, we used quan i a i e measu es o spa ial
au oco ela ion o e eal egions o he seaoo whe e geog aphic clus e ing o mega auna was
s a is ically signican (beyond wha would be expec ed om andom chance). In his con ex ,
spa ial au oco ela ion quan ies he deg ee o which he abundance o mega auna in a gi en
sampling uni is simila o he a e age abundances o neighbou ing sampling uni s. Thus, he
choice o he op imal neighbou hood size is key because i di ec ly inuences he ou come and
subsequen in e p e a ion o ho spo analysis esul s: o e ly la ge neighbou hood sizes may smoo h
away local spa ial pa e ns, whe eas o e ly small neighbou hoods may be e y sensi i e o noise and
o he spu ious a e ac s in he abundance da a ma ix. He e, we dened ou op imal
neighbou hood size o comp ise six nea es neighbou s, a e empi ically obse ing ha o mos
di es, he a e change in spa ial au oco ela ion (Mo an’s I) does no change signican ly om
a ound he six h-o de neighbou hood. (Figu e 5).
Figu e 5: Spa ial co ela ion alues plo ed agains die en sizes o k- h o de neighbou hoods.
A ule o humb o choosing he op imal neighbou hood size o ho spo analysis is o look o
he inec ion poin o he cu e, which ep esen s a balance be ween oo ew and oo many
neighbou s.
To de ec mega aunal ho spo s and coldspo s, we s used k-nea es neighbou algo i hm41 o
cons uc a g aph ha connec s each sampling uni o i s six nea es neighbou s (based on
geog aphic p oximi y). Based on his neighbou hood g aph, we calcula ed Local Indica o s o
Spa ial Associa ion (LISA) s a is ics o each sampling uni , which iden ied localised egions
whe e mega aunal abundances we e signican ly highe o lowe han would be expec ed om
spa ial andomness. To classi y hese geog aphic clus e s (as ei he ho spo s o coldspo s), we
p ojec ed he LISA s a is ics on o a Mo an’s sca e plo (Supplemen a y Figu e S2), which shows
he ela ionship be ween he abundance o each sampling uni e sus he a e age abundances o i s
neighbou s (spa ial lag). Depending on whe e a gi en sampling uni was loca ed on his sca e plo ,
we classied i as ei he a ho spo (high-high abundances), coldspo (low-low abundances) o an
ou lie (low-high o high-low abundances). Finally, we assessed he s a is ical signicance o he
obse ed spa ial pa e ns (o LISA s a is ics) by conduc ing a andomised hypo hesis es unde he
null hypo hesis o comple e spa ial andomness.
Assessing mega aunal biodi e si y and communi y composi ion
Ben hic biodi e si y assessmen s a e key owa ds unde s anding he o e all ecosys em heal h and
unc ionali y. He e, we used s anda d de ia ion o mega aunal abundances and Shannon di e si y
index o measu e di e si y in bo h he Eas e n and Wes e n egion. This egional compa ison o
di e si y allowed us o simul aneously assess bo h spa ial a iabili y and dep h-wise zona ion
pa e ns o mega auna, since he Eas e n and Wes e n egions a y by wa e dep h and dis ance o
sho e (wi h dis inc die ences in ca bon expo o he seaoo , upwelling p ocesses, and inpu o
e igenous ma e ial).
We iden ied clus e s o mega aunal communi ies using non-pa ame ic mul i a ia e s a is ics.
Fi s , we applied double- oo ans o ma ion o he abundance da a ma ix o s abilise he a iance
and mode a e he inuence o dominan axa (abundance da a ypically con ains many low alues
and ew high alues). Nex , we used he ans o med abundances o gene a e a B ay-Cu is
simila i y ma ix ha cap u es he deg ee o bio ic (dis)simila i y among he sampling uni s. We
hen applied hie a chical agglome a i e clus e ing (wi h g oup-a e age linking) o his simila i y
ma ix, he eby e ealing clus e s o sampling uni s wi h simila bio ic composi ion.
To o mally es whe he he die ences among he majo clus e s o mega aunal communi ies was
s a is ically signican , we pe o med an analysis o simila i y (ANOSIM). ANOSIM calcula es a
es s a is ic R ha cap u es he a e age die ence be ween in e - and in a-communi y simila i ies,
wi h he null hypo hesis H0 dened as: The e is no significan diffe ence in bio ic composi ion
among he main mega aunal communi ies. In addi ion, we used a simila i y pe cen ages analysis
(SIMPER) o iden i y axa ha con ibu ed he mos owa ds he sepa a ion among espec i e
clus e s o mega aunal communi ies.
Finally, we isualised he in e - ela ionships among mega aunal communi ies by p ojec ing he
sampling uni s on o a wo-dimensional o dina ion ( ea u e) space using non-me ic
mul idimensional scaling (nm-MDS). We also supe imposed on o he o dina ion plo axa and
en i onmen al a iables ha we sampled a he cen oids o espec i e sampling uni s. These
abio ic a iables included: dep h, slope, opog aphic posi ion index, e ain uggedness, salini y,
empe a u e and longi ude. This g aphical ep esen a ion allows o a con enien isual
in e p e a ion o he associa ion be ween bio ic and abio ic a iables, oge he wi h hei inuence
on he iden ied mega aunal communi ies. (No e ha we omi ed la i ude since all ou
deploymen s we e along an Eas -Wes ansec . Also, longi ude he e is p opo ional o dis ance
om sho e bu no o dep h, e en hough he wo a iables a e co ela ed o some ex en ).
Resul s
This sec ion p esen s ndings om ou unsupe ised seaoo classica ion, along wi h a
desc ip ion o mega aunal axa ha we de ec ed in he a ea. We u he desc ibe he spa ial
dis ibu ion pa e ns o hese mega auna, as well as en i onmen al d i e s ha inuence hei
dis ibu ion in bo h Eas e n and Wes e n egions.
Visibili y imp o emen
Figu e 6 shows quali a i e esul s o ou colou co ec ion wo kow. The educ ion in image
in ensi y owa ds he edges is now accoun ed o , and he o e all con as is enhanced in he
ans o med image. The co ec ion also emo es he g eenish haze ha was p e alen in he aw
images, esul ing in good dis ibu ion o colou s o e he en i e enhanced image. Collec i ely, hese
ans o ma ions p oduce well-illumina ed scenes ha e eal bio a and subs a e cha ac e is ics wi h
clea con as e.g. highligh ing animal acks and sedimen dis u bance due o biogenic ac i i ies.
Figu e 6: Examples showing isibili y imp o emen ans o ma ion om (A) o iginal images, o
(B) colou no malised images. The mega aunal axa, animal ails and bio u ba ion-d i en
sedimen dis u bance a e now clea ly isible in he ans o med images.
Seaoo subs a e classica ion
Figu e 7 shows esul s o ou unsupe ised classica ion o he seaoo . The classica ion is based
on he ex ac ed isual in o ma ion o he en i e image, which encodes bo h biogenic and
abiogenic p ope ies o he pho og aphed seabed (e.g. bio u ba ion, lebensspu en, bu ows,
seaoo mo phology, sedimen colou , e ce e a). Each poin co esponds o an indi idual image
mapped in ea u e space, while colou coding is based on he wel e seaoo classes assigned o he
espec i e images using unsupe ised K-means classica ion.
Figu e 7: P ojec ion o images (as poin s) in ea u e space, colou coded by one o 12 seaoo
subs a e classes. Images om espec i e di es clus e oge he because hey ep esen he same
geog aphic egion on he seabed and hus ha e mos simila sedimen ological and ben hic
p ope ies. The images also show sub-pa i ions wi hin di es, which is an indica ion o sub le
die ences in seabed subs a es a small spa ial scales. O e all, he e is a clea a iabili y o PC1
ha links o inc easing in ensi y o sedimen dis u bance by bio u ba ing o ganisms (highe PC1
alues).
Sedimen sampling du ing c uise M182 showed ha he seaoo is composed o so sedimen o
die en g ain size and composi ion. Towa ds he Eas and in close p oximi y o land, he amoun
o ne g ained (sil ) e igenous sedimen inc eases, while owa ds he Wes sedimen s a e s ongly
domina ed by o amini e a shells. The clus e ing shows ha he seabed exhibi s sub le die ences
a small ange along a su ey ansec , while he e we e clea die ences a egional scale sepa a ing
he die en su eys om each o he .
By manually inspec ing subsample images om each su ey-clus e (Figu e 8), we obse ed ha
hese die ences eec ed he ex en o sedimen dis u bance by biogenic ac i i ies (on eeding and
mo ing acks), and in he sedimen (bu ow holes, sedimen mounds). The dis u bance was mos ly
p onounced in he ex u e o images om he shallowe Eas e n egion (su eys 131, 144, 145),
cha ac e ised by bu ows, pi s and ose e-like s uc u es esembling a sweeping polychae e a m.
The ea u e space p ojec ion cap u es his die ence, by showing a le - o- igh (Wes o Eas )
g adien in e ms o bio u ba ion in ensi y.
Figu e 8: Image subsamples showing ha a iabili y in subs a e cha ac e is ics be ween he
Eas e n and Wes e n egions was inuenced by he deg ee o sedimen dis u bance om biogenic
ac i i y.
Mega aunal abundance and di e si y
Ou FaunD-Fas model de ec ed 10189 mega aunal o ganisms belonging o 13 axa g oups. To
check o po en ial double coun ing o mega auna due o o e lapping images, we compa ed he
a e age dis ance be ween successi e images agains he a e age leng h o he along- ack image axis
(o ien ed in he di ec ion o image acquisi ion). The esul s o his compa ison a e shown in Figu e
9, whe e we only ound o e lap in di e 19 ou o he se en di es. No e ha di e 19 was also whe e
he sampling equency was highes (0.2 Hz).
Figu e 9: Rela ionship be ween a e age dis ance be ween successi e images and he a e age image
leng h in he di ec ion o image acquisi ion. Fo a gi en di e, he e was o e lap i he image
leng h was sho e han he dis ance be ween images.
Figu e 13: Cho ople h maps showing he dis ibu ion o mega aunal abundances along
espec i e su ey ansec s. O e all, he di es in he shallow eas e n egion exhibi high
abundance consis en ly along ansec s whe eas he abundances a e low in he deepe Wes e n
egion, excep in opog aphically complex habi a s like on op o seamoun s.
Figu e 14 shows a ep esen a ion o ho spo s, coldspo s and ou lie s ha we plo ed in p ole iew.
Compa ed o he cho ople h map abo e, only loca ions wi h s a is ically signican clus e s o
high/low mega aunal abundances ( ela i e o local neighbou hoods) a e colou coded.
Figu e 14: P ole iew o geog aphic clus e ing showing he dis ibu ion o s a is ically
signican ho spo , coldspo s and ou lie s along espec i e ansec s. The size o he symbol is
p opo ional o he mega aunal abundance in he co esponding sampling uni a ha loca ion.
A ou chosen scale o analysis (100 me es) and small neighbou hood size (o 6), he gu e
shows ha ho spo s o mega auna a e p edominan ly ound in complex opog aphic ea u es e.g
he op o seamoun (di e 32), abyssal hills (di e 28) and on he sides o a subma ine canyon (di e
144).
The Eas e n egion was cha ac e ised by a s a is ically signican ho spo o high mega aunal
abundances a he s a o he s eep side o he subma ine canyon in di e 144, wi h he o he hal o
he c oss sec ion exhibi ing coldspo s o ela i ely lowe abundances. Con a y o expec a ions, we
did no obse e s a is ically signican geog aphic clus e s on he seamoun in di e 145, po en ially
because he seabed he e was unde sampled due o came a mal unc ion (No e he sho e leng h o
di e 145). In he Wes e n egion, s a is ically signican ho spo s o mega auna we e obse ed in
complex physical landscapes e.g on op o he seamoun (in di e 32) as well as on op o local abyssal
hills (in di e 28). Sho s e ches o coldspo s we e loca ed in ela i ely a abyssal plains (a he
s a o di e 78 and a he end o di e 28). We did no obse e signican ou lie s.
Inuence o bio ic and abio ic d i e s
Figu e 15 shows he o dina ion o all he 232 sampling uni s p ojec ed on o a wo-dimensional
nm-MDS ea u e space. The 64 sampling uni s om he Eas e n egion mos ly g oup oge he in o
a small igh clus e /communi y on he ex eme le o he o dina ion space, whe eas he emaining
168 sampling uni s om he Wes e n egion a e sca e ed h oughou (al hough hey span mos ly
he igh hal o he ea u e space). A ew smalle sub-clus e s a e also isible in he Wes e n egion.
In e ms o bio ic/communi y composi ion, he axa ha p edominan ly inuenced he deepe
Wes e n egion include Echinode ma a, Fo amini e a, and A h opoda. The emaining majo i y o
axa p edominan ly inuenced he shallowe Eas e n egion, which may explain he high numbe o
biogenic s uc u es (Lebensspu en) ha we obse ed in he Eas e n egion.
Supe imposing en i onmen al a iables on o he o dina ion plo shows ha he ho izon al axis
dis inguishes mega aunal communi ies based on empe a u e and dep h a iables. This is ob ious
conside ing ha he wo a iables map close o he bounda y sepa a ing communi ies in he
Eas e n and Wes e n egions. On he o he hand, ba hyme ic de i a i es such as slope, uggedness,
oughness and posi ioning index p edominan ly inuenced mega aunal communi ies a g ea
dep hs in he Wes e n egion.
Figu e 15: P ojec ion o sampling uni s on o nm-MDS o dina ion space. Colou coding is based
on he di es whe eas he symbols dis inguish be ween he wo egions. Also supe imposed in he
o dina ion plo a e axa and en i onmen al d i e s ha po en ially s uc u e mega aunal
communi ies in he wo egions. I is clea ha he e is a dis inc ion be ween he wo main
mega aunal communi ies in he Eas e n and Wes e n egion, and also ha ba hyme ic d i e s
p edominan ly s uc u e communi ies in he deepe Wes e n egion.
Discussion
So a , we ha e demons a ed ha inco po a ing A.I in o ma ine science wo kows accele a es he
cha ac e isa ion o habi a s and mega auna dis ibu ion in deep sea en i onmen s. He e, we
p o ide u he in e p e a ions o he obse ed pa e ns, and con ex ualise he ndings ela i e o
o he s udies.
Ou semi-supe ised seaoo sedimen classica ion wo kow in ol ed clus e ing based on
encoded isual ea u es, ollowed by manual in e p e a ion o he clus e s o assign seman ic
meaning. We chose his app oach because mode n implemen a ions o he K-means clus e ing
algo i hm a e as , accu a e and s aigh o wa d o use 42, he eby enabling au oma ed so ing o
huge olumes o unlabelled images in o manageable ep esen a i e g oupings. In conduc ing
ben hic sedimen mapping o he Aus alian Na ional Ma ine Bio egionalisa ion p ojec , Luciee
e al. 43 also poin ou ha s a is ical clus e ing allows o mo e objec i i y and epea abili y in
image-based seaoo classica ion when compa ed o manual in e p e a ion. In ano he s udy,
Diesing e al. 44 emphasise he key ole o ea u e space p ojec ion me hods such as p incipal
componen s analysis (PCA) owa ds enabling isual in e p e abili y o seaoo clus e s. Clus e ing
also ensu es consis ency in seaoo anno a ion since isually simila images a e almos gua an eed
o be assigned he same labels, as was also poin ed ou by La h op e al. in p e ious ben hic habi a
cha ac e isa ion s udy in New Yo k Bigh 45. Howe e , he clus e ing pe o mance will depend on
he me hod used o encode isual in o ma ion om images: hand-enginee ed ea u es like ex u e
a e bes o ep esen ing ob iously he e ogeneous seabed e.g as was p e iously used o cha ac e ise
he dis ibu ion o Mn-nodules in he Cla ion Clippe on Zone 9 46. Fo his s udy, we ex ac ed
high-le el ea u es using a p e- ained con olu ional neu al ne wo k ha ha e been shown in pas
s udies e.g by Yamada e al. 47 o be capable o cap u ing sub le a iabili y in seabed subs a e
composi ion. Colou -coding he ea u e space using clus e ing labels p oduces a g aphical display
ha allows o a quick (quali a i e) isual s imp ession o sedimen cha ac e is ics. This kind o
display p o es use ul o decision making by ma ine scien is s du ing expedi ions e.g o de e mine
whe e o sample nex , o o help in he choice o an app op ia e image da ase o s udying a gi en
phenomenon o in e es (e.g om eposi o ies like BIIGLE 48 o PANGAEA 49). We used his
g aphical display in ea u e space as ou main in e ace o semi-au oma ed anno a ion because (a) i
was mo e con enien o assign seman ic labels o clus e s compa ed o labelling indi idual images,
(b) i was easy o le e age con ex ual in o ma ion e.g cha ac e is ics o neighbou ing clus e s o
adap he anno a ion acco dingly based on he unde lying s uc u e o he da a, and (c) i was
s aigh o wa d o de ec any anomalous pa e ns and/o a e ac s as hese clus e s would be
unusually isola ed in he ea u e space. In app oaching seaoo classica ion his way, we conside
au oma ed algo i hms o be use ul agen s o p elimina y so ing, while ese ing he nal call o
anno a o s wi h he domain expe ise o esol e nuanced, g anula and sub le a iabili y in
subs a e composi ion ha may be missed by algo i hms.
Deploying ou FaunD-Fas model o de ec mega aunal axa p oduced weak anno a ions ha s ill
needed o be manually inspec ed, ened and e-labeled, as was also done p e iously by Tang e al. 50
and Zhang e al 51. In ou case, he model was able o co ec ly de ec ins ances o mega auna in
images (wi h an accu acy o 78.1%), e en hough axa labels assigned o he de ec ions we e
some imes inco ec e en o o ganisms ha we e p esen in bo h he A lan ic and he Pacic.
While o e ing and poo gene alisa ion may explain he misclassica ion o p e iously unseen
axa 24, i is no ob ious why some o ganisms (e.g Holo hu ians and Ophiu oids) ha we e p esen
in bo h he Pacic and A lan ic we e co ec ly de ec ed ye misclassied, despi e colou no malising
he wo da ase s in he same way. A possible explana ion is oe ed by Zu owie z e al. 52 who
p e iously poin ed ou ha concep d i poses a big challenge o knowledge ans e ac oss
ma ine en i onmen s, especially in s udies in ol ing non-endemic axa. In his con ex , concep
d i is he phenomenon whe e he s a is ical dis ibu ion o isual p ope ies o ma ine o ganisms
shi s ac oss da ase s ei he g adually o suddenly, as was also highligh ed by Langenkämpe e al. 53.
The e o e, exac ly how o de elop a species de ec ion model ha gene alises ac oss oceans emains a
challenging open p oblem ha needs u he in es iga ion. In p inciple, such a gene alizable model
mus be al oge he agnos ic o he dis inc die ences in e ms o ecological habi a s, in a- and
in e -species appea ance, wa e column p ope ies e ce e a. Recen ly, he e ha e been eo s aimed
a add essing hese gene alisa ion challenges by de eloping well-cu a ed s anda dised benchma k
image da ase s. Fo example, he openly a ailable global image da abase Fa homNe by Ka ija e al.
54 p o ides anno a ions ha co e a wide ange o axa ca ego ies om die en ocean
en i onmen s. The goal o hese benchma k da ase s is o enable aining and e alua ion o deep
lea ning models ha would be mo e obus and gene alizable, since he models would ha e been
exposed o di e se isual ea u es o ma ine o ganisms 53. Ano he po en ial solu ion o poo
gene alisa ion is da a augmen a ion, which in ol es he applica ion o andom geome ic and
pho ome ic ans o ma ions e.g andom scaling, o a ions and ipping in o de o a icially
inc ease he olume and a ie y o aining examples, as has p e iously been demons a ed by Tan
e al. 55. How well hese (and o he ) solu ions wo k is a p omising di ec ion o u he esea ch.
Despi e he a o emen ioned challenges, ou p e- ained FaunD-Fas model is s ill di ec ly use ul in
si ua ions whe e one only ca es abou bina y auna/non- auna de ec ions e.g. o dis inguish ma ine
o ganisms om o he backg ound objec s in a li e OFOS/ROV ideo eed 56 57.
Absolu e mega aunal axa coun s ha we ob ained om ou auna anno a ion wo kow equi ed
s anda disa ion due o po en ial sampling bias and lack o a consis en (spa ial) scale o e e ence.
Howe e , he choice o an op imal leng h (o esolu ion) o he sampling uni wi hin which o
s anda dise he anno a ions is no ob ious bu depends on he p oblem and ecological con ex , as
was also p e iously poin ed ou by En iche i e al. 20 and Dominguez-Ca i e al. 58 In ou case, we
chose a xed-size leng h o 100 me es because we we e in e es ed in cap u ing localised mega aunal
dis ibu ion pa e ns in high esolu ion, along linea ansec s whose a iabili y in subs a e
cha ac e is ics was e y sub le. Acco ding o guidelines om a p e ious s udy on ansec s and
quad a s in ecology by Mu ay e al. 59, we conside ou 100-me e-long sampling uni s o be high
esolu ion conside ing ha he a e age leng h o ou ansec s was 3200 me es. Mu ay e al 59
ecommended he use o high esolu ion sampling uni s whene e possible (and esou ce
pe mi ing), since he ne esolu ion allows o he cap u ing o g anula localised dis ibu ion
pa e ns e.g mega auna adap ed o mic ohabi a s, specic dep h g adien s o subs a e ype o e
sho dis ances. Choosing la ge -sized sampling uni s (e.g wi h esolu ions o 500 me es o g ea e )
may a e age ou small scale spa io-ecological pa e ns, and a e bes sui ed o p o iding gene alised
in o ma ion ega ding o e all ends in communi y s uc u e e.g as was p e iously a gued by
Mon aña e al 60. In any case, we su eyed he seaoo a sucien ly high equency (maximum 0.2
Hz), esul ing in an a e age o 38 images wi hin each o ou 232 sampling uni s o each 100m
leng h. This sample size is sucien o unbiased and obus biodi e si y assessmen s using
mul i a ia e s a is ics, as has p e iously been demons a ed by Fo cino e al. 61. Ou chosen
esolu ion was also con enien pu ely om a compu a ional pe spec i e 62, because isualising he
232 sampling uni s in he o dina ion ea u e space did no equi e oo much memo y o compu ing
esou ces.
The e we e majo die ences in he dis ibu ion and abundance o mega auna be ween he Eas e n
and Wes e n egions. As Ramos e al. also poin ou in hei p e ious s udy o ma ine biodi e si y
o Mau i anian deep wa e s 63, he egional die ences in abundance may be explainable by
a iabili y in dep h and geomo phological complexi y o he seabed. Conside ing ha he a e age
dep h die ence be ween he Eas e n and Wes e n egions o ou wo king a ea was app oxima ely
700 me es, he high mega aunal abundances and di e si y in he shallowe Eas e n egion migh be
due o ood a ailabili y in he o m o sinking o ganic ma e 64. Since he Eas e n egion is also
ela i ely close o Mau i anian sho e, he egion bene s mo e om bo h land-based nu ien
sou ces as well as om nu ien en ichmen om upwelling cu en s, as has also been p e iously
epo ed by Scepanki e al. 65. Mo eo e , ou CTD p oles in Supplemen a y Figu e S1 show ha
he shallowe wa e s in he Eas e n egion also exhibi ela i ely wa me empe a u es (2.85°C ±
0.12) compa ed o he Wes e n egion (2.50°C ± 0.08). These wa me empe a u es may also ha e
con ibu ed o he obse ed high mega aunal abundance in he Eas e n egion, since ele a ed
empe a u es ha e been shown o enhance me abolic a es while also suppo ing a wide ange o
unc ional ai s, e.g as shown by Pue a e al. 66 and Swee man e al. 67. Bo h he ansec dep h
p oles and he hill-shaded ba hyme ic g id (Figu e 2) show he p esence o s uc u ally complex
ea u es like seamoun s and local ele a ions in he Eas e n and Wes e n egions. In gene al, we
obse ed high mega aunal abundances in hese complex habi a s compa ed o a e ains because
he complex opog aphies c ea e mic ohabi a s e.g ocky ou c ops and sedimen pocke s, which
p o ide shel e and p o ec ion o mega auna while also inuencing hyd odynamic condi ions o
c ea e s onge cu en s ha p omo e nu ien dis ibu ion and iche ood webs 68. S ill, we
obse ed highe abundances on seamoun s in he Eas e n egion compa ed o hose in he deepe
Wes e n egion, which could be because POC ux is highe in shallow seamoun s due o s onge
upwelling eec s, as was also p e iously shown by Vic o e o e al. 69. Rega ding he inuence o
en i onmen al d i e s on he obse ed spa ial pa e ns, ou o dina ion plo showed ha
ba hyme ic d i e s such as slope, uggedness, and oughness p edominan ly inuenced mega aunal
communi ies in he deepe Wes e n egion. This could be because habi a s in he deepe seabed
a eas a e in gene al mo e s able, wi h opog aphies ha a y slowly in kilome e scale 70. As a esul ,
e en mino a iabili y in he ba hyme ic de i a i es in he deepe seabed esul s in a mo e
p onounced inuence on hyd odynamic eec s like cu en pa e ns and nu ien dis ibu ion.
This is in con as o he al eady complex opog aphies in shallowe pa s ha na u ally dis up
hyd odynamic ows, so ha he eec o mino changes in ba hyme ic de i a i es a e no as
p onounced e.g as was also poin ed ou by Kaise e al. 71.
Collec i ely, he abo e ndings demons a e ha inco po a ion o A.I in o con en ional
image-based ma ine science wo kows does con ibu e owa ds expedi ed cha ac e isa ion o
subs a e ypes and mega aunal dis ibu ion as ollows: Fi s he e a e ob ious speed gains since
human eo is equi ed o seman ically label only he A.I gene a ed anno a ions ins ead o he
en i e da ase . Second, ou wo kow p ac ically demons a es how p ojec ing seaoo images on o
ea u e space does allow o a quick a -a-glance isualisa ion (and in e p e a ion) o na u al ben hic
habi a g oupings, including any anomalies ha equi e u he in es iga ions. Thi d, ou pu s
om ou A.I wo kows (e.g da a ma ices) seamlessly in eg a e wi h exis ing classical
spa io-ecological wo kows such as o dina ion and spa ial au oco ela ion analysis. This
in eg a ion allows us o au oma e only he necessa y epe i i e ime-consuming asks (like
anno a ion), while a oiding unnecessa y e-in en ion o he wheel in downs eam analysis. Fou h,
despi e he occasional misclassica ions, ou model does gene alise ac oss oceans o he ex en ha
i co ec ly de ec s and localises o ganisms in images. This is di ec ly use ul in applica ions such as
apid unde wa e ideo analysis o e.g ex ac ele an ames o subsequen seman ic labelling.
The e o e, we conclude ha au oma ed image analysis wo kows ha e he capaci y o ecien ly
ex ac ac ionable insigh s om e aby e-scale seaoo image y, which is necessa y o complemen
bo h ongoing and planned de elopmen o imely ma ine baseline in o ma ion o moni o ing
emo e ben hic ecosys ems a egional and global scale.
Acknowledgemen s
We acknowledge ha he seaoo images used in his esea ch we e acqui ed du ing RV METEOR
c uise M182, unde he amewo k o he REEBUS p ojec ha is unded h ough BMBF g an
03F0815. We app ecia e he ope a o s o he XOFOS sys em on boa d he essel, and o all he c ew
and scien is s who made he da a acquisi ion campaign success ul. We also acknowledge Daphne
Cu elie o manually inspec ing and assigning seman ic axa labels o he weak mega auna
anno a ions. The s au ho wan s o specically hank he Helmhol z School o Ma ine Da a
Science (Ma DATA), G an No. HIDSS-0005, o di ec nancial suppo . All au ho s decla e ha
his esea ch was conduc ed in he absence o any comme cial o nancial ela ionships ha could
be cons ued as a po en ial conic o in e es . This is publica ion 65 o he DeepSea Moni o ing
G oup a GEOMAR Helmhol z Cen e o Ocean Resea ch Kiel.
Au ho con ibu ion s a emen
B.M concei ed, implemen ed and p og ammed he compu e ision and spa io-ecological analysis
wo kows, and also d a ed he manusc ip . J.G was he chie scien is in c uise M182, and also
pa icipa ed in he concep ion, design, and o e all coo dina ion o he esea ch, as well as p o iding
geospa ial in e p e a ions and d a ing he manusc ip . All au ho s ead and app o ed he nal
manusc ip .
Da a a ailabili y s a emen
The da ase s p esen ed in his s udy can be ound online in BIIGLE he e
h ps://anno a e.geoma .de/p ojec s/65. In e media e da a gene a ed du ing he analysis is also
p o ided in he supplemen a y ma e ials as an excel le.
To eques he da a used in his s udy, please con ac P o . D . Jens G eine (jg eine @geoma .de)
Re e ences
1. Dano a o, R., Snelg o e, P. V. R. & Tyle , P. Challenging he pa adigms o deep-sea ecology.
T ends Ecol. E ol. 29, 465–475 (2014).
2. Riehl, T., Wöl, A.-C., Augus in, N., De ey, C. W. & B and , A. Disco e y o widely a ailable
abyssal ock pa ches e eals o e looked habi a ype and p omp s e hinking deep-sea
biodi e si y. P oc. Na l. Acad. Sci. 117, 15450–15459 (2020).
3. Webb, T. J., Be ghe, E. V. & O’Do , R. Biodi e si y’s Big We Sec e : The Global Dis ibu ion
o Ma ine Biological Reco ds Re eals Ch onic Unde -Explo a ion o he Deep Pelagic Ocean.
PLOS ONE 5, e10223 (2010).
4. Mon es, E. e al. Op imizing La ge-Scale Biodi e si y Sampling Eo : Towa d an Unbalanced
Su ey Design. Oceanog aphy 34, 80–91 (2021).
5. Howell, K. L. e al. A Bluep in o an Inclusi e, Global Deep-Sea Ocean Decade Field
P og am. F on . Ma . Sci. 7, (2020).
6. Dumke, I. e al. Unde wa e hype spec al imaging as an in si u axonomic ool o deep-sea
mega auna. Sci. Rep. 8, 12860 (2018).
7. Sala, E. & Knowl on, N. Global Ma ine Biodi e si y T ends. Annu. Re . En i on. Resou . 31,
93–122 (2006).
8. Hu enne, V. A. I. e al. ROVs and AUVs. in Subma ine Geomo phology (eds. Micalle , A.,
K as el, S. & Sa ini, A.) 93–108 (Sp inge In e na ional Publishing, Cham, 2018).
doi:10.1007/978-3-319-57852-1_7.
9. Mbani, B., Schoening, T., Gazis, I.-Z., Koch, R. & G eine , J. Implemen a ion o an
au oma ed wo kow o image-based seaoo classica ion wi h examples om
manganese-nodule co e ed seabed a eas in he Cen al Pacic Ocean. Sci. Rep. 12, 15338
(2022).
10. Ma in Lud igsen, Johnsen, G., Sø ensen, A. J., Lågs ad, P. A. & Ødegå d, Ø. Scien ic
Ope a ions Combining ROV and AUV in he T ondheim Fjo d. Ma . Technol. Soc. J. 48,
59–71 (2014).
11. Moni uzzaman, Md., Islam, S. M. S., Bennamoun, M. & La e y, P. Deep Lea ning on
Unde wa e Ma ine Objec De ec ion: A Su ey. in Ad anced Concep s o In elligen Vision
Sys ems (eds. Blanc-Talon, J., Penne, R., Philips, W., Popescu, D. & Scheunde s, P.) 150–160
(Sp inge In e na ional Publishing, Cham, 2017). doi:10.1007/978-3-319-70353-4_13.
12. Du, Y., Liu, Z., Li, J. & Zhao, W. X. A Su ey o Vision-Language P e-T ained Models.
a Xi .o g h ps://a xi .o g/abs/2202.10936 2 (2022).
13. Deng, J. e al. ImageNe : A la ge-scale hie a chical image da abase. in 2009 IEEE Con e ence on