GigaScience , 2023, 12 , 1–12
DOI: 10.1093/gigascience/giad062
Technical No e
BigSeqKi : a pa allel Big Da a oolki o p ocess FASTA
and FASTQ iles a scale
Césa Piñei o and Juan C. Pichel *
CiTIUS, Uni e sidade de San iago de Compos ela, San iago de Compos ela 15782, Spain
∗Co espondence add ess . J uan C . Pichel, CiTIUS , Uni e sidade de San ia go de Compos ela, San ia go de Compos ela 15782, Spain. E-mail: juanca los.pic [email p o ec ed]
Abs ac
Bac kg ound: High- h oughpu sequencing echnologies ha e led o an unp eceden ed explosion in he amoun s o sequencing da a
a aila b le , whic h a e ypicall y s o ed using F ASTA and F ASTQ iles. W e can ind in he li e a u e se e al ools o p ocess and manipula e
hose ype o iles wi h he aim o ans o ming sequence da a in o biolog ical knowledg e . How e e , none o hem a e well i ed
o p ocessing e icien ly e y la ge iles, likely in he o de o e aby es in he ollowing yea s, since hey a e based on sequen ial
p ocessing. Onl y some ou ines o he well-known seqki ool a e pa ly pa allelized. In any case, i s scalabili y is limi ed o use ew
h eads on a single compu ing node.
Resul s: Ou app oach, BigSeqKi , akes ad an a ge o a high-pe o mance compu ing–Big Da a amewo k o pa allelize and op imize
he commands included in seqki wi h he aim o speeding up he manipula ion o F ASTA/F ASTQ iles. In his way, in mos cases, i
is om ens o hund eds o imes as e han se e al s a e-o - he-a ools. A he same ime, ou oolki is easy o use and ins all on
any kind o ha dw a e pla o m (local se e o clus e ), and i s ou ines can be used as a bioin o ma ics lib a y o om he command
line.
Conclusions: BigSeqKi is a e y comple e and ul a- as oolki o p ocess and manipula e la ge F ASTA and F ASTQ iles. I is pub licl y
a aila b le a h ps://gi hub.com/ci iususc/BigSeqKi .
Ke yw o ds: F ASTA/F ASTQ iles, Pe o mance, Pa allelism, Big Da a
In oduc ion
The his o y o mode n DNA sequencing s a ed se e al decades
ago and, since hen, has seen as ounding g ow h in sequencing
capaci y and speed. F om he i s genomes wi h a ew housand
bases, DNA sequencing has ad anced o sequence he human
genome o 3 billion bases. In ecen yea s, nex -gene a ion se-
quencing (NGS) echnology, also known as massi e pa allel se-
quencing (MPS), has made i possible o expand he amoun o
sequencing da a a ailable . Fo example , he Illumina No aSeq
6000 [ 1 ] pla o m can gene a e a maximum ou pu o 6 Tb o da a
and ead abou 20 billion sequences pe un. No e ha sequences,
commonl y named eads , a e composed o ASCII c ha ac e s ep e-
sen ing a nucleo ide (base) om he sequence. In he DNA case,
we can only ind 4 possible bases (A—adenine , C—cy osine , G—
guanine, and T— hymine).
T he NGS a w da a a e mainly s o ed in FASTA [ 2 ] and FASTQ [ 3 ]
ex -based ile o ma s. In pa icula , nucleo ide and p o ein se-
quences a e ypically s o ed in he FASTA ile o ma , whe eas
FASTQ is he mos widely used o ma o sequencing ead da a.
An example o FASTA ile is shown in Fig. 1 . A sequence in FASTA
o ma begins wi h a single-line desc ip ion abou he sequence
in he subsequen lines . T he desc ip ion line is dis inguished om
he sequence da a by a g ea e - han ( > ) symbol a he beginning.
On he o he hand, he FASTQ o ma was designed o handle he
quali y me ics o he sequences ob ained om he sequence s. In
FASTQ, e e y 4 lines desc ibe a sequence o ead. An example is
displa yed in Fig. 2 . T he in o ma ion p o ided pe ead is as ollows:
iden i ie and an op ional desc ip ion ( i s line), sequence (second
line), and he quali y sco e o he ead ( ou h line). An ex a ield,
ep esen ed by symbol “+ ,” is used as sepa a o be ween he da a
and he quali y in o ma ion ( hi d line).
Manipula ing hese iles e icien ly is essen ial o analyze and
in e p e da a in any genomics pipeline. Common ope a ions
on F ASTA and F ASTQ iles include sea ching, il e ing, sampling,
deduplica ion, and so ing, among o he s. We can ind se e al
ools in he li e a u e o FASTA/Q ile manipula ion such as HT-
Seq [ 5 ], FASTX [ 6 ], q ools [ 7 ], seq k [ 8 ], Biopy hon [ 9 ], sam ools [ 10 ],
p y adix [ 11 ], p y as x [ 12 ], and seqki [ 13 ]. These ools can be clas-
si ied acco ding o how he sequences a e pa sed [ 12 ]. In he
i s ca ego y, sequences a e p ocessed in o de , whic h causes im-
po an o e heads when ex ac ing and andomly sampling se-
quences . T ha is he case o HTSeq , FASTX , q ools , and seq k . In
he second ca ego y, we ind ools ha suppo andom access o
sequences by es ablishing an index ile. Tools belonging o his
ca ego y a e mo e e icien in e ms o pe o mance and memo y
consump ion. Ho w e e , none o hem a e well i ed o p ocess-
ing e y la ge iles o hund eds o GB (likel y TBs in he nea u u e)
since hey a e based on sequen ial p ocessing. The excep ion is
seqki ha allows some ou ines o use a ew h eads, bu in any
case, i s scalabili y is e y limi ed.
To deal wi h his issue, in his a icle, we in oduce BigSeqKi ,
a pa allel oolki o manipula e FASTA and FASTQ iles a scale
wi h speed and scalabili y a i s co e. BigSeqKi akes ad an a ge
o IgnisHPC [ 14 , 15 ], a compu ing engine ha uni ies he de elop-
men , combina ion, and execu ion o high-pe o mance compu -
ing (HPC) and Big Da a pa allel asks using di e en languages
Recei ed: Ma c h 7, 2023. Re ised: May 25, 2023. Accep ed: July 10, 2023
©The Au ho (s) 2023. Published by Ox o d Uni e si y P ess GigaScience. This is an Open Access a icle dis ibu ed unde he e ms o he C ea i e Commons
A ibu ion License ( h ps://c ea i ecommons.o g/licenses/by/4.0/ ), which pe mi s un es ic ed euse, dis ibu ion, and ep oduc ion in any medium, p o ided
he o iginal wo k is p ope l y ci ed.
Downloaded om h ps://academic.oup.com/gigascience/a icle/doi/10.1093/gigascience/giad062/7233988 by UDC - Uni e sidade da Co una use on 26 Oc obe 2023
2 | GigaScience , 2023, Vol. 12, No. 1
Figu e 1: Example o FASTA ile showing he i s pa o he PAX6 gene (ob ained om [ 4 ]).
Figu e 2: Example o FASTQ ile o ma (ob ained om [ 4 ]).
and p og amming models. As i was demons a ed, IgnisHPC ou -
pe o ms he s a e-o - he-a ame wo k Spa k [ 16 ] in e ms o
pe o mance and scalabili y unning applica ions ha ep esen
he mos ypical algo i hmic pa e ns in Big Da a and scien i ic
compu ing.
BigSeqKi uses he seqki ou ines as basis since ha oolki co -
e s a wide ange o u ili ies and is one o he mos used by he
bioin o ma ics esea c h comm uni y. As a consequence, BigSeqKi
will o e he same unc ionali ies and command in e ace [ 17 ].
BigSeqKi can be used om he command line, bu i is a he same
ime a lib a y, so i s ou ines can also be called om a C/C ++ ,
Py hon, Go, o Ja a applica ion.
Ano he impo an c ha ac e is ic o BigSeqKi is ha i is ully
con aine ized, which isola es he execu ion en i onmen om he
physical sys em and a oids dependenc y p oblems. As a conse-
quence, BigSeqKi is e y easy o ins all and can un on a local
se e o on any ype o clus e since i suppo s some o he mos
impo an esou ce and schedule manage s (e.g., Mesos [ 18 ], No-
mad [ 19 ], and Slu m [ 20 ]).
Bac kg ound
IgnisHPC [ 14 , 15 ] uni ies he execu ion o Big Da a and HPC wo k-
loads in he same compu ing engine. Unlike o he ame wo ks
such as Hadoop [ 21 ] and Spa k [ 16 ], IgnisHPC has na i e suppo
o m ul ilangua ge a pplica ions using bo h JVM (Ja a Vi ual Ma-
chine) and non-JVM-based languages. In his way, applica ions can
be implemen ed using 1 o se e al p og amming langua ges ol-
lowing an API inspi ed by Spa k’s one.
The p e ious e sion o IgnisHPC suppo ed na i el y C, C ++ ,
J a a, and Py hon. Ho w e e , seqki w as implemen ed using he
Go p og amming langua ge. Since BigSeqKi pa allelizes and op-
imizes he seqki ou ines using IgnisHPC, i was necessa y o
add suppo o his language in he ame wo k. Ano he solu-
ion would equi e o po he comple e oolki o a di e en lan-
gua ge, whic h is a di icul ask p one o e o s. I is wo h no -
ing ha , o he bes o ou kno wledge, no w adays IgnisHPC is
he i s pa allel compu ing amewo k o include na i e sup-
po o his language. Conside ing Spa k ins ead o IgnisHPC
is no an op ion because, as i was demons a ed in [ 14 ], when
using a nonna i e language code, da a ans e s be ween he
JVM and ex e nal p ocesses deg ade no iceabl y Spa k’s o e all
pe o mance.
Go is a p og amming langua ge wi h a simple syn ax ha was
designed o be easy o lea n and use. Wi h he elease o Go 1.18,
he language included suppo o Gene ics, which allows he c e-
a ion o unc ions , ypes , and me hods ha can wo k wi h any
da a ype . T his mak es Go an e ec i e and use - iendl y way o
implemen Big Da a in e aces . T he implemen a ion o Go in Ig-
nisHPC is simila o ha o C ++ , as bo h a e compiled and s a i-
cally yped languages. Ho w e e , Go eplaces he concep o inhe -
i ance wi h composi ion, which does no change he philosophy
o use in IgnisHPC. Big Da a unc ions a e s ill accessible h ough
he IgnisHPC API, and use s can c ea e hei own code by imple-
men ing he same in e aces.
One o he k e y ea u es o IgnisHPC is i s use o con aine s o
isola e and execu e code. Con aine s a e ligh weigh and po able,
making i easy o un IgnisHPC on a a ie y o di e en clus e s,
including bo h HPC and Big Da a. IgnisHPC is also ole an o ail-
u es, as he con aine s o p ocesses can be easil y es a ed i he e
a e issues. In pa icula , i some da a a e los , IgnisHPC has enough
in o ma ion abou how i was de i ed. In his way, only hose op-
e a ions needed o ecompu e he co esponding po ion o da a
a e pe o med.
We mus highligh ha al hough he IgnisHPC API [ 22 ] uses a
sequen ial no a ion, ope a ions on da a a e pe o med in pa allel.
As we poin ed ou , he IgnisHPC API was inspi ed by he Spa k
API in such a way ha IgnisHPC codes a e easily unde s and-
able by use s who a e amilia wi h Spa k. Table 1 shows a lis o
some o he mos impo an unc ions suppo ed by IgnisHPC. In
pa icula :
Map unc ions: The common c ha ac e is ic o ou ines belong-
ing o his ype is ha hey a ppl y he same unc ion o each
elemen in he da a. As a esul o he ans o ma ion, he
ou pu could be o di e en size wi h espec o he inpu .
Reduce unc ions: educe and eeReduce me hods a gg ega e
all he elemen s in he inpu da a using a unc ion. agg e-
ga e and eeAgg ega e a e a so o educ ion whe e he
ype o he inpu and ou pu da a is di e en . In his case,
2 unc ions a e necessa y; he i s one is applied o each
Downloaded om h ps://academic.oup.com/gigascience/a icle/doi/10.1093/gigascience/giad062/7233988 by UDC - Uni e sidade da Co una use on 26 Oc obe 2023
BigSeqKi | 3
Table 1: Some o he mos impo an IgnisHPC API unc ions
Type Func ions
Map map, la map, mapWi hIndex, il e , keyBy,
keys, alues, mapPa i ions, mapValues, e c.
Reduce educe, eeReduce, agg ega e, eeAgg ega e,
educeByKey, agg ega eByKey, e c.
G oup g oupBy, g oupByKey
So so , so By, so ByKey
I/O pa allelize,collec , op, ake,
sa eAsObjec File, sa eAsTex File,
sa eAsJsonFile, e c
SQL union, join, dis inc
Ma h sample, sampleByKey, ake, akeSample, coun ,
coun ByKey, coun ByValue, max, min, e c.
Balancing epa i ion, pa i ionByHash,
pa i ionByRandom, pa i ionBy
Pe sis ence pe sis , cache, unpe sis , uncache
elemen in a da a pa i ion, and he second one combines
he pa ial esul s ob ained o each pa i ion. educeByKey
and agg ega eByKey a e a ia ions whe e he ope a ion is
pe o med only among elemen s wi h he same k e y in such
a way ha he inal esul is a se o unique pai s wi h
alues calcula ed using educe o agg ega e ope a ions,
espec i el y.
G oup unc ions: These me hods g oup elemen s in a da a
ame acco ding o hei k e y alue ( g oupByKey ) o a use -
de ined unc ion ( g oupBy ).
So unc ions: In o de o so elemen s , IgnisHPC p o ides 3
unc ions: so , so ByKey , and so By . The i s me hod
uses he na u al o de and does no need any addi ional
unc ion. so ByKey so s he k e ys using hei na u al o de .
so By allows o use a use -de ined unc ion o speci y he o -
de o he elemen s. I he esul o applying ha unc ion o
2 elemen s is ue , hen he i s elemen should p ecede he
second one. All me hods suppo ascending and descending
o de .
SQL unc ions: These unc ions ope a e on da a ames. union
conca ena es 2 da a ames, join me ges elemen s o 2 da a
ames whose k e ys ma ch, and dis inc e u ns a new da a
ame a e emo ing he duplica e eco ds . T hese me hods
a e necessa y, o example, in many g aph p ocessing p ob-
lems.
O he unc ions: IgnisHPC implemen s se e al ope a ions ha
e u n a alue o he d i e code, bu hey do no modi y o
gene a e ne w s o ed da a. Spa k e e s o his ype o ope a-
ions as ac ions . Fo ins ance, IgnisHPC suppo s me hods such
as coun , ake , akeSample , and collec . The mos basic
ope a ion is coun ha e u ns he numbe o elemen s o a
s o ed da a collec ion. collec e u ns a collec ion wi h all
he elemen s s o ed in he execu o s o a ask. ake applies
a collec ope a ion bu ob ains only he i s n elemen s,
whe e n is chosen by he use . akeSample e u ns a an-
dom sample o n elemen s om he dis ibu ed da a, wi h
o wi hou eplacemen . Finall y, ano he in e es ing ou ine
is pa allelize , which dis ibu es he elemen s o a collec-
ion among he execu o s o o m a dis ibu ed da ase . In
his case, new s o ed da a a e c ea ed.
I is wo h no ing ha he IgnisHPC API unc ions allow use s
o pa allelize a code wi h a high le el o abs ac ion. In his way,
i is only necessa y o ocus on da a dependencies.
Table 2: Lis o commands included in bo h BigSeqKi and seqki .
Those commands wi h an as e isk suppo new unc ionali ies
no included in seqki
Basic commands
seq T ans o m sequences (ex ac ID, il e by leng h,
emo e ga ps, e e se complemen , e c.)
subseq Ge subsequences by egion/g /bed, including
lanking sequences
s a s Simple s a is ics o FASTA/Q iles: #seqs, min/max
leng h, N50, Q20%, Q30%, e c.
aidx
∗C ea e FASTA o FASTQ index ile and ex ac
subsequences
Fo ma con e sion
a2 q Re ie e co esponding FASTQ eco ds b y a FASTA
ile
q2 a Con e FASTQ ile o FASTA o ma
ansla e T ansla e DN A/RN A o p o ein sequence
Sea ching
g ep Sea ch sequences by
ID/name/sequence/sequence mo i s
loca e Loca e subsequences/mo i s
Se ope a ions
sample Sample sequences by numbe o p opo ion
mdup Remo e duplica ed sequences by
ID/name/sequence
common Find common sequences o mul iple iles by
ID/name/sequence
duplica e Duplica e sequences N imes
head P in i s N FASTA/Q eco ds
head-genome P in sequences o he i s genome wi h common
p e ixes in name
pai Ma ch up pai ed-end eads om 2 FASTQ iles
ange P in FASTA/Q eco ds in a ange (s a :end)
Edi
conca Conca ena e sequences wi h he same ID om
mul iple iles
eplace Replace name/sequence using a egula
exp ession
ename Rename duplica ed IDs
O de ing
so So sequences by ID/name/sequence/leng h
shu le Shu le sequences
Me hods
As we commen ed p e iousl y, BigSeqKi ( RRID:SCR _ 023592 ) speeds
up he seqki ou ines h ough pa alleliza ion and op imiza ion
echniques. Table 2 shows he ou ines suppo ed by he cu en
e sion o BigSeqKi . Despi e mos o he commands in seqki a e
sequen ial, we can classi y each command implemen a ion in o
3 ca ego ies acco ding o i s inhe en pa allelism:
Independen : i is a emba assingl y pa allel wo kload. As a
consequence, he compu a ion could be applied o all se-
quences in pa allel. An example is seq , a unc ion ha ans-
o ms sequences. In his case, he ans o ma ion onl y a ec s
each sequence indi idually.
P a iall y dependen : compu a ions could be done in pa allel,
bu he me hod equi es some ype o consensus o ob ain he
esul . Fo ins ance, s a s should me ge he pa ial esul s
compu ed o each sequence o calcula e some s a is ics o
he conside ed FASTA/Q ile.
De penden : de pendencies be ween sequences p e en he
me hod om being execu ed in pa allel. As a consequence,
Downloaded om h ps://academic.oup.com/gigascience/a icle/doi/10.1093/gigascience/giad062/7233988 by UDC - Uni e sidade da Co una use on 26 Oc obe 2023
4 | GigaScience , 2023, Vol. 12, No. 1
BigSeqKi equi es a comple e ne w algo i hm o pe o m
he same command in pa allel. mdup is a good example
because wi h he aim o emo ing duplica ed sequences,
i is necessa y o ead all o hem be o e gene a ing a
esul .
The in eg a ion, pa alleliza ion, and op imiza ion o each seqki
command in IgnisHPC will be di e en depending on i s ca ego y.
Mo e de ails a e p o ided below.
Independen ou ines
Fo hese commands, he compu a ion can be applied o all se-
quences in pa allel because he e a e no dependencies (commu-
nica ion) among hem. In o he w o ds, ou ines belonging o his
ca ego y can be p ocessed using an emba assingly pa allel ap-
p oac h. Conside ing he IgnisHPC (and Spa k) API, i is only nec-
essa y o use map unc ions o pa allelize he compu a ions. As
we poin ed ou , he common c ha ac e is ic o hese API unc-
ions is ha hey a ppl y he same ope a ion o each elemen in he
da a.
The ollowing BigSeqKi commands belong o his ca ego y: seq ,
subseq , s a s , q2 a , a2 q , ansla e , g ep , loca e , dupli-
ca e , and eplace (see Table 2 o de ails).
Pa ially dependen ou ines
As we men ioned, his ca ego y includes commands in which
compu a ions can be done in pa allel using map unc ions, bu
he me hods equi e some ype o consensus o ge he desi ed
ou come . T his consensus can be easily implemen ed using he
IgnisHPC API. The ollowing BigSeqKi commands belong o his
ca ego y:
s a s : s a is ics can be gene a ed in pa allel bu he i-
nal esul m us be unique, so all pa ial esul s m us be
me ged using a educ ion ( educe ope a ion in he IgnisHPC
API).
head : sequences should know hei posi ion inside he ile
o c hec k i hey a e inside he head window . T o do ha , i is
necessa y o use mapWi hIndex , a special map ope a ion in-
cluded in he IgnisHPC API ha allows each elemen o know
i s global index wi hin a da a s uc u e.
head-genome : simila o head , bu no all sequences a e alid.
In o de o de e mine he window, in alid sequences mus be
emo ed i s .
ange : also simila o head . Sequences should know hei
posi ion inside he ile o c hec k i hey a e wi hin he ange
window.
g ep : al hough his command was included in he p e ious
ca ego y, a command op ion ( --dele e-ma ched ) limi s he
numbe o esul s o jus 1 pe sea ch pa e n. In such cases,
i is necessa y o emo e he ex a esul s.
aidx : also simila o head , sequences compu e hei
o se s inside he inpu ile using mapPa i ionWi hIn-
dex and exchange he in o ma ion be ween execu o s
o pe o m a pa allel indexing ope a ion wi h a simple
map .
Dependen ou ines
Commands belonging o his ca ego y ha e an implemen a ion in
seqki ha by i s na u e canno be pa allelized. Ho w e e , IgnisHPC
allows us o de ine he implemen a ion a a high le el, whic h in-
c eases no iceabl y he p oduc i i y. Beha io s and unc ionali ies
will be p ese ed in BigSeqKi bu h ough a comple e ne w pa allel
implemen a ion. In pa icula :
sample : a sequen ial sampling can be pe o med in pa allel i
we spli he sequences and un a sample o each pa i ion. I
was ma hema ically p o en ha sampling wi hou eplace-
men ollows a hype geome ic unc ion [ 23 ]. In his way, we
can calcula e he p opo ion o he sample ha co esponds
o each pa i ion.
mdup : sequences a e g ouped ( g oupBy API unc ion) using
a hash wi h he ID, name, o sequence. In hose g oups con-
aining mo e han 1 elemen , a sea ch o duplica es is ca ied
ou o emo e hem.
pai and conca : sequences o he inpu iles gene a e k e y–
alue pai s whe e he k e y is he ID and he alue is he se-
quence wi h i s index ile ( map ). P ai s a e uni ied by means o
union and g ouped using g oupByKey . A e w a d, sequences
in he same g oup a e pai ed o conca ena ed i hey belong
o di e en iles.
common : he i s s age o he command is he same one ex-
plained abo e o pai and conca . Then i a sequence can be
ound in all iles, we c hec k i s index ile, o a oid i s dele ion.
ename : sequences a e g ouped ( g oupBy ) using hei ID, and
hen IDs in he same g oup a e enamed.
so : he sequen ial so algo i hm implemen ed in seqki is
eplaced by a sample Me geSo [ 24 ] algo i hm ha can be
e icien ly execu ed in pa allel in a dis ibu ed en i onmen .
shu le : sequences shu ling can be implemen ed using he
IgnisHPC API unc ion pa i ionByRandom .
Ano he implemen a ion de ails
In o de o pa allelize and in eg a e he seqki ou ines in o Ig-
nisHPC, i was necessa y o s a conside ing he sequence pa se .
I akes a s eam o c ha ac e s in F ASTA and F ASTQ o ma and
gene a es a da a s uc u e wi h he sequence ep esen a ion. In
seqki , his s eam can be ep esen ed by a ile o he s anda d in-
pu . In BigSeqKi , his s eam is implemen ed using he IgnisHPC
i e a o s, whic h g an he use s access o he da a pa i ions. In
his way, BigSeqKi will ead he da a om a ile and spli i in
m ul iple pa i ions, whic h acili a es hei pa allel p ocessing. In
pa icula , each wo ke eads a po ion o he inpu ile, so he
inpu /ou pu (I/O) ope a ion is pe o med in pa allel. The e is 1
wo ke pe compu ing node. Wi hin each wo ke , i s po ion o he
ile is u he di ided among he a ailable h eads , imp o ing he
o e all I/O pe o mance. As a esul , he seqki command a gu-
men s ha a ec ile p ocessing will ha e no e ec in BigSeqKi .
Fo example, he -- wo-pass op ion, which eads a ile mul iple
imes ins ead o s o ing all he sequences in memo y, does no
make sense in BigSeqKi . We mus highligh ha he ac o spli -
ing he inpu iles be ween se e al compu ing nodes in BigSeqKi
means ha he memo y consumed by node is also spli , which
allows ou ool o wo k wi h la ge da ase s. In addi ion, BigSeqKi
also educes he memo y oo p in by only s o ing he IDs and
indices o each sequence.
Ano he impo an ad an a ge o using IgnisHPC is how mem-
o y is handled. Use s can choose a ype o s o a ge acco ding o
hei pa icula case. Fo ins ance, i an inpu ile is oo la ge
o be k e p comple ely in he se e memo y, i could be s o ed
comp essed in memo y o in disk. Pe o mance would be lowe ,
bu i could be success ull y p ocessed. Tha scena io is no
conside ed by seqki ha simply would aise an “ou o mem-
o y” e o . In pa icula , BigSeqKi suppo s he ollowing s o a ge
op ions:
Downloaded om h ps://academic.oup.com/gigascience/a icle/doi/10.1093/gigascience/giad062/7233988 by UDC - Uni e sidade da Co una use on 26 Oc obe 2023
BigSeqKi | 5
In-Memo y: i is he bes pe o me since all da a a e s o ed in
memo y. I is he de aul op ion.
Raw memo y: da a a e s o ed in a memo y bu e using a se i-
alized bina y o ma . Ex a memo y consump ion is minimal
and he bu e is comp essed by Zlib.
Disk: simila o aw memo y bu he bu e is s o ed as a
POSIX ile. Al hough he pe o mance is signi ican ly wo se,
i enables wo king wi h as amoun s o da a ha canno be
en i el y k e p in memo y.
On he o he hand, mdup , common , and pai commands in
seqki use hash unc ions o c hec k duplica es. I is well known
ha hash unc ions can p oduce he same esul o di e en
alues . T his e en is commonly known as a hash collision. How-
e e , seqki does no c hec k o collisions, so i is possible o
gene a e inco ec esul s. BigSeqKi uses hashes o g oup se-
quences bu hen c hec ks o collisions by compa ing he eal
alues.
Finally, seqki and o he s a e-o - he-a ools build index iles
( aidx ou ine) o speed up some o he asks (e.g., sea ches). Al-
hough BigSeqKi is also capable o c ea ing hose index iles, i
does no equi e hem o imp o e i s pe o mance since da a
wi hin IgnisHPC a e al eady indexed. In o he w o ds, he index
is c ea ed while eading he inpu ile.
New unc ionali ies
BigSeqKi no only enables he pa alleliza ion o seqki unc ions
bu also imp o es i s algo i hms o p o ide bene i s e en o se-
quen ial execu ions and includes addi ional unc ionali ies. In
pa icula , he aidx command in seqki implemen s indexing
o FASTA iles using he sam ools o ma , bu FASTQ iles a e
no suppo ed. BigSeqKi adds suppo o his ype o iles and
gene a es an index ile using he sam ools o ma as well. No e
ha his is he mos widesp ead o ma and is also suppo ed
by o he s a e-o - he-a ools . T he e o e, BigSeqKi allows index-
ing o bo h FASTA and FASTQ iles using he same syn ax han
seqki .
How o Use BigSeqKi
BigSeqKi can be used in 2 di e en wa ys . T he i s one is by means
o a command-line in e ace (CLI). This a pp oac h is simila o he
“command subcommand” s uc u e adop ed by seqki [ 13 ]. In his
way, i is only necessa y o selec a subcommand o ou ine (see
a comple e lis in Table 2 ) and pass i s a gumen s h ough com-
mand line. As we men ioned p e iousl y, o imp o e he usabili y
and acili a e he adop ion o BigSeqKi , i implemen s he same
command in e ace as seqki .
Since BigSeqKi uns wi hin he IgnisHPC ame wo k, i is neces-
sa y o send he BigSeqKi ou ine h ough he IgnisHPC submi e .
Fo ins ance, i we a e unning BigSeqKi on a local se e , he ol-
lowing exp ession uses he ou ine seq o p in he name o he
sequences included in a FASTA ile o an ou pu ile:
The e o e, he syn ax should be: ignis-submi ig-
nishpc/ ull bigseqki < cmd > < a gumen s > .
In addi ion, use s can also speci y h ough a gumen s he num-
be o ins ances , co es , and memo y (in GB) o be used in he exe-
cu ion. By de aul , hose alues a e se o 1. Fo example, we can
execu e he p e ious command using 2 co es:
Figu e 3:
Example o Py hon code using he BigSeqKi ou ines.
Unlike he o he s a e-o - he-a ools, BigSeqKi can also
be execu ed on a pa allel clus e . Typical HPC clus e s ha e
Slu m [ 20 ] as he p e e ed esou ce mana ge and Singula i y [ 25 ]
as a con aine -based echnology. In his case, use s will send
he job using he ignis-slu m submi e ins ead o ignis-
submi .
On he o he hand, BigSeqKi can also be used as a bioin o ma -
ics lib a y. I is wo h no ing ha BigSeqKi was implemen ed in Go
language. Ho w e e , hanks o he mul ilanguage suppo p o ided
by IgnisHPC, i is possible o call BigSeqKi ou ines om C/C ++ ,
Py hon, Ja a, and Go applica ions wi hou addi ional o e head.
An example o Py hon code is shown in Fig. 3 . This example is
equi alen o he p e ious one used in he explana ion o he CLI.
Since BigSeqKi has been c ea ed as a lib a y, i only needs o be im-
po ed o be used. Func ions in BigSeqKi do no use iles as inpu ;
hey use Da aF ames ins ead, an abs ac ep esen a ion o pa al-
lel da a used by IgnisHPC (simila o RDDs in Spa k). P a ame e s
a e g ouped in a da a s uc u e whe e eac h ield ep esen s he
long names o a pa ame e . We mus highligh ha BigSeqKi unc-
ions can be link ed (lik e sys em pipes using “| ”), so he Da aF ame
gene a ed by one can be used as inpu o ano he . In his way,
in eg a ing BigSeqKi ou ines in a mo e complex code is eally
easy.
The code s a s ini ializing he IgnisHPC ame wo k (line 5 in
Fig. 3 ). Nex , a clus e o con aine s is con igu ed and buil (lines
om 7 o 15). Mul iple pa ame e s can be used o con igu e he en-
i onmen suc h as ima ge , numbe o con aine s , numbe o co es ,
and memo y pe con aine . In his example, we will use 1 node
(ins ances) and 2 co es by node. A e con igu ing he IgnisHPC ex-
ecu ion en i onmen , he BigSeqKi code ac ually s a s. Fi s , we
ead he inpu ile (line 17). The e is a di e en unc ion o ead-
ing F ASTA and F ASTQ iles. All he inpu sequences a e s o ed as a
single da a s uc u e . T he nex s age consis s o p in ing he name
o he sequences included in he FASTA ile (line 19). The unc ion
Downloaded om h ps://academic.oup.com/gigascience/a icle/doi/10.1093/gigascience/giad062/7233988 by UDC - Uni e sidade da Co una use on 26 Oc obe 2023

6 | GigaScience , 2023, Vol. 12, No. 1
akes as pa ame e s he sequences and he op ions ha speci y
i s beha io . Finally, he names o he sequences a e w i en o
disk. I is impo an o highligh ha lazy e alua ion is pe o med,
so unc ions a e onl y execu ed when he esul is equi ed o be
sa ed on disk.
Expe imen al Resul s
In his sec ion, we analyze he pe o mance esul s ob ained by
BigSeqKi wi h espec o o he s a e-o - he-a ools. In pa ic-
ula , we ha e conside ed sam ools , py as x , and seqki o hei
pe o mance and numbe o commands suppo ed. Expe imen s
we e conduc ed using up o 8 compu ing nodes o he FinisT e -
ae III [ 26 ] supe compu e ins alled a CESGA (Spain). Each node
con ains a 32-co e In el Xeon Ice Lake 8352Y @2.2 GHz p oces-
so and 256 GB o memo y in e connec ed wi h In iniband HDR
100. I is a Linux clus e unning Rocky Linux 8.4 (ke nel 4.18.0).
We ha e used Singula i yCE 3.9.7 (con aine s), IgnisHPC 2.2, py-
as x 0.8.4, sam ools 1.16.1, and seqki 2.3.1 (wi h Slu m as clus-
e manage and Lus e as dis ibu ed ile sys em).
The pe o mance e alua ion was ca ied ou using as inpu 6
di e en F ASTA/F ASTQ iles ha co e a wide a ie y o c ha ac e -
is ics and sizes . T he main ea u es o hese iles a e he ollowing:
D
1 ( m64013e_210227_222017.hi i_ eads —FASTA—
24 GB):
Numbe o sequences: 1.2M, Minimum leng h: 85, A e age
leng h: 19.7K, Maximum leng h: 48.5K.
D
2
( SRR642648_1. il —FASTQ—24.1 GB):
Numbe o sequences: 98.7M, Minimum leng h: 100, A e a ge
leng h: 100, Maximum leng h: 100.
D
3 ( Homo_sapiens.GRCh38.dna_sm. ople el —FASTA—
59.7 GB):
Numbe o sequences: 639, Minimum leng h: 970, A e a ge
leng h: 98.8M, Maximum leng h: 248.9M.
D
4
( ERR4667750 —FASTQ—79.1 GB):
Numbe o sequences: 318.1M, Minimum leng h: 101, A e a ge
leng h: 101, Maximum leng h: 101.
D
5
( unip o _ embl —FASTA—104 GB):
Numbe o sequences: 229.9M, Minimum leng h: 7, A e a ge
leng h: 351.6, Maximum leng h: 45.3K.
D
6
( DRR002180_2 —FASTQ—395 GB):
Numbe o sequences: 1.625B, Minimum leng h: 101, A e a ge
leng h: 101, Maximum leng h: 101.
As example o illus a e he bene i s o ou ool, we will e alu-
a e he ollowing u ili ies (see Table 2 o a comple e lis o com-
mands): aidx builds an index o F ASTA/F ASTQ iles, loca e
loca es sequences ollowing some sea ch pa e n, eplace e-
places a name/sequence using a egula exp ession, mdup e-
mo es duplica ed sequences , sample selec s sequences by num-
be o p opo ion, seq ans o ms sequences (ex ac ID, il e
by leng h, e c.) and emo es ga ps, and so so s sequences
by ID/name/sequence/leng h. We will also include he pe o -
mance esul s o he co esponding u ili ies, i hey exis , o sam-
ools , py as x , and seqki . Execu ion imes o all he ools con-
side ed include he o e head o loading sequences in o mem-
o y and he subsequen w i ing o esul s o disk. No e ha he
“2-pass” a gumen o seqki was no used in he expe imen s.
Eac h esul was compu ed as he median o 5 expe imen s. Fo
he sake o ep oducibili y, all he codes and sc ip s used o
pe o ming he benchma ks a e eely a ailable a he BigSeqKi
eposi o y.
Fi s , in o de o p o ide an o e all idea abou he scalabil-
i y and pe o mance o BigSeqKi wi h espec o he o he s a e-
o - he-a ools, we will show he speedups ob ained o he D
4
da ase using di e en numbe o co es . T he beha io is e y sim-
ila when conside ing he o he da ase s. Resul s in log scale a e
displayed in Fig. 4 . Speedups we e calcula ed using as e e ence
he sequen ial execu ion (1 co e) o he co esponding BigSeqKi
command. Acco ding o he esul s, se e al conclusions can be
made. I can be obse ed ha he scalabili y o BigSeqKi is qui e
good, eac hing speedups up o 27.7 ×and 95.7 ×( seq command)
using 1 se e (32 co es) and 8 compu ing nodes (256 co es), e-
spec i el y. No e ha speedups o some ou ines a e no highe
when using 256 co es due o a small ac ion o he code ha
should be execu ed sequen ially (Amdahl’s law).
While sam ools and py as x ou ines a e always p ocessed se-
quen ially, seqki uses a mul i h eaded app oach o (pa ly) pa -
allelize some commands. Ho w e e , i s scalabili y is limi ed o
use a ew h eads on a single se e (compu ing node). This
is he case o loca e . I s bes speedup onl y eac hes 11.3 ×(32
co es) while his alue inc eases un il 19.6 ×wi h BigSeqKi . I
8 nodes a e used, BigSeqKi is 49.9 × as e han he sequen ial
execu ion.
Fo all he commands s udied, BigSeqKi clea ly ou pe o ms
sam ools , py as x , and seqki . The e a e onl y a e w cases using 1 co e
whe e he speedups o hese ools a e sligh ly g ea e han 1 (e.g.,
execu ing he aidx ou ine wi h sam ools and py as x ). Ho w e e ,
o he commands such as so and sample a e p ocessed as e
wi h BigSeqKi e en using 1 co e.
Tables om 3 o 9 display, o all he da ase s, he execu ion
imes o BigSeqKi and he o he s a e-o - he-a ools when un-
ning aidx , loca e , eplace , mdup , sample , seq , and so u il-
i ies, espec i el y. Speedups wi h espec o he sequen ial exe-
cu ion o he co esponding BigSeqKi command a e shown be-
ween b ac ke s . Highligh ed is he as es ime o e all and he
co esponding speedup. No e ha BigSeqKi s o es comp essed
in memo y he la ges da ase D
6 when using 1 compu ing
node since i exceeds he memo y capaci y o an indi idual
se e (see he Raw memo y s o age op ion in he Backg ound
sec ion).
Fo all he expe imen s conduc ed, BigSeqKi is always he
as es ool bo h conside ing a single se e (1 node) o se e al
compu ing nodes. In any case, le ’s ake a look in de ail o he be-
ha io o each command:
aidx (Table 3 ): BigSeqKi speedups ange om 5.4 × o 27.4 ×
conside ing a single se e (32 co es) and om 7.2 × o 144 ×
wi h 8 nodes. I means, o example, building he index ile
o ou la ges da ase D
6 (395 GB) in jus 5.8 minu es (sin-
gle se e ), while sam ools and py as x equi e abou 2.1 hou s.
This ime dec eases o 1 minu e when BigSeqKi uses 8 nodes.
As men ioned p e iousl y, he aidx ou ine in seqki does no
suppo FASTQ iles (D
2
, D
4
, and D
6
).
loca e (Table 4 ): he sea ching ou ines, g ep and loca e ,
a e e y expensi e in e ms o compu a ions. No e ha
conside ing sequen ial p ocessing, loca e akes mo e han
3 hou s o p ocess ou smalles da ase D
1
independen ly o
he ool conside ed. This ime inc eases o mo e han 3 days
o compu a ion o D
6
. seqki has a m ul i h ead e sion o
loca e , which ob ains speedups om 10.5 × o 18.8 ×. These
speedups a e always lowe han he ones ob ained by BigSe-
qKi on a single se e . I is impo an o highligh ha seqki
aises an ou -o -memo y e o when p ocessing D
6
wi h 1, 2,
and 4 co es. On he o he hand, when using 8 nodes, BigSe-
Downloaded om h ps://academic.oup.com/gigascience/a icle/doi/10.1093/gigascience/giad062/7233988 by UDC - Uni e sidade da Co una use on 26 Oc obe 2023
BigSeqKi | 7
Figu e 4: Speedups (in log scale) ob ained by BigSeqKi and o he s a e-o - he-a ools wi h espec o he BigSeqKi sequen ial ime when execu ing
di e en commands using D
4
as inpu . No e ha loca e was pa allelized in seqki .
qKi ac hie es no iceable speedups up o 104.1 ×. In his way, i
is able o educe he ime necessa y o execu e he loca e
command wi h ou la ges da ase D
6 om 3 days o only
0.8 hou s.
eplace (Table 5 ): his ou ine (o an equi alen ) is no
suppo ed by sam ools and py as x . In his case, BigSeqKi is
om ens o hund eds o imes as e han seqki , eac hing
speedups up o 159.8 ×.
mdup (Table 6 ): his ou ine is also no suppo ed by sam ools
and py as x . In his case, BigSeqKi is ens o imes as e han
seqki , ac hie ing a maxim um speedup o 74.7 ×when emo -
ing he duplica ed sequences in D
5
.
sample (Table 7 ): ope a ion no suppo ed by sam ools . BigSe-
qKi is again as e han he o he ools, inc easing he
speedups as he inpu da a size g ows. I can be obse ed
ha BigSeqKi is able o sample sequences in seconds. Fo in-
s ance , p y as x and seqki ake abou 3 hou s o p ocess D
6
,
while BigSeqKi equi es jus 2 minu es.
seq (Table 8 ): ope a ion no suppo ed by sam ools . Pe o -
mance esul s a e simila o he sample ones in such a way
ha BigSeqKi il e s sequences by ID in a ew seconds, achie -
ing a no iceable speedup o 169.7 ×. I should be no ed ha
among he ou ines examined in his s udy, seq is he leas
compu a ionally demanding.
so (Table 9 ): his ou ine was no included in py as x . In
gene al, he pe o mance o sam ools and seqki is poo . And,
mos impo an ly, bo h ools p oduce memo y e o s when
p ocessing he la ges da ase D
6
, so i canno be so ed. How-
e e , BigSeqKi so s D
6
21.8 ×and 131.1 × as e han he se-
quen ial e sion using a single se e and 8 compu ing nodes,
espec i el y. I means ha he ime dec eases om 5 hou s o
ba el y 2 minu es.
Finall y, we m us highligh ha one o he main easons o he
di e ences in he speedups be ween da ase s unning he same
command wi h BigSeqKi is he load balance be ween h eads. I
will depend on he c ha ac e is ics o he da ase : numbe o se-
quences and hei leng h.
Conclusions
Cu en s a e-o - he-a ools such as seqki , py as x , and sam ools
a e no eady o p ocessing and manipula ing e y la ge FASTA
and FASTQ iles because all o hem a e mainl y based on sequen-
ial p ocessing. To ha end, we ha e p esen ed BigSeqKi , which
pa allelizes and op imizes he seqki ou ines using he IgnisHPC
compu ing ame w o k. Since seqki w as p og ammed in Go, Ig-
nisHPC was ex ended o suppo ha language. As a consequence,
IgnisHPC is no w adays he i s pa allel compu ing ame wo k ha
suppo s Go. BigSeqKi can be easily ins alled on a local se e o
on a clus e . In addi ion, i can be used om he command line
o as a lib a y. Thanks o he mul ilanguage suppo o IgnisHPC,
BigSeqKi ou ines can be called om C/C ++ , Py hon, Ja a, and Go
codes.
Downloaded om h ps://academic.oup.com/gigascience/a icle/doi/10.1093/gigascience/giad062/7233988 by UDC - Uni e sidade da Co una use on 26 Oc obe 2023
8 | GigaScience , 2023, Vol. 12, No. 1
Table 3: Execu ion imes (seconds) using di e en numbe o co es: aidx command. Highligh ed a e as es ime and numbe o imes
as e han sequen ial BigSeqKi
1 2 4 8 16 32 64 (2 nodes) 128 (4 nodes) 256 (8 nodes)
D
1
sam ools 86.2 [1.03 ×] – – – – – – – –
py as x 109.2 [0.81 ×] – – – – – – – –
seqki 75.4 [1.17 ×] – – – – – – – –
BigSeqKi 88.4 46.0 35.3 26.3 19.4 16.3 [5.4 ×] 13.6 12.3 [7.2 ×] 12.5
D
2
sam ools 165.6 [1.06 ×] – – – – – – – –
py as x 177.9 [0.99 ×] – – – – – – – –
BigSeqKi 175.9 90.8 67.4 50.3 39.1 31.4 [5.6 ×] 23.4 19.1 15.5 [11.3 ×]
D
3
sam ools 210.0 [0.77 ×] – – – – – – – –
py as x 131.2 [1.23 ×] – – – – – – – –
seqki 131.8 [1.23 ×] – – – – – – – –
BigSeqKi 161.9 83.9 61.7 24.5 17.5 15.7 [10.3 ×] 13.6 13.4 [12.1 ×] 14.7
D
4
sam ools 538.4 [1.27 ×] – – – – – – – –
py as x 615.5 [1.11 ×] – – – – – – – –
BigSeqKi 684.2 346.6 175.2 90.3 45.4 29.3 [23.3 ×] 19.6 15.3 12.5 [54.7 ×]
D
5
sam ools 771.0 [1.08 ×] – – – – – – – –
py as x 634.3 [1.31 ×] – – – – – – – –
seqki 1,096.2 [0.76 ×] – – – – – – – –
BigSeqKi 829.8 361.3 179.4 89.3 49.4 30.3 [27.4 ×] 23.6 19.3 16.5 [50.3 ×]
D
6
sam ools 7,651.6 [1.14 ×] – – – – – – – –
py as x 7,712.5 [1.13 ×] – – – – – – – –
BigSeqKi 8,712.3 4,423.3 2,282.2 1,191.9 640.2 350.4 [24.9 ×] 129.5 85.3 60.5 [144 ×]
Table 4: Execu ion imes (seconds) using di e en numbe o co es: loca e command. Highligh ed a e as es ime and numbe o imes
as e han sequen ial BigSeqKi
1 2 4 8 16 32 64 (2 nodes) 128 (4 nodes) 256 (8 nodes)
D
1
py as x 11,523.5 [1.0 ×] – – – – – – – –
seqki 12,822.9 6,385.0 3,210.9 1,731.4 940.5 612.4 [18.8 ×] – – –
BigSeqKi 11,486.2 6,286.1 3,180.0 1,637.3 850.9 470.6 [24.4 ×] 264.6 156.9 110.3 [104.1 ×]
D
2
py as x 8,841.2 [1.2 ×] – – – – – – – –
seqki 12,319.8 6,909.4 3,335.9 1,746.2 997.3 971.2 [10.5 ×] – – –
BigSeqKi 10,168.6 5,264.5 2,711.5 1,412.2 814.6 545.4 [18.6 ×] 384.7 293.5 234.9 [43.3 ×]
D
3
py as x 13,075.3 [1.1 ×] – – – – – – – –
seqki 14,281.6 8,161.7 5,009.6 3,184.1 1,832.4 1,054.9 [14.1 ×] – – –
BigSeqKi 14,834.2 8,223.3 4,572.8 2,585.6 1,494.6 872.1 [17.0 ×] 532.8 365.9 262.5 [56.5 ×]
D
4
py as x 30,028.3 [1.05 ×] – – – – – – – –
seqki 39,640.5 21,257.6 10,803.1 5,715.1 3,369.7 2,795.2 [11.3 ×] – – –
BigSeqKi 31,615.2 16,832.1 8,531.9 4,433.3 2,466.8 1,609.9 [19.6 ×] 1,074.7 794.6 633.5 [49.9 ×]
D
5
py as x 27,876.5 [1.06 ×] – – – – – – – –
seqki 31,301.8 16,884.7 9,141.1 4,698.4 2,971.8 2,802.9 [10.5 ×] – – –
BigSeqKi 29,540.7 15,431.3 8,120.2 4,401.4 2,454.5 1,443.9 [20.5 ×] 908.1 599.5 440.9 [67 ×]
D
6
py as x 270,214 [1.02 ×] – – – – – – – –
seqki Ou o Mem. Ou o Mem. Ou o Mem. 40,122 23,075 18,309 [15.0 ×] – – –
BigSeqKi 275,680 141,095 72,110 37,140 19,810 11,477 [24.0 ×] 7,003 4,422 3,080 [89.5 ×]
Downloaded om h ps://academic.oup.com/gigascience/a icle/doi/10.1093/gigascience/giad062/7233988 by UDC - Uni e sidade da Co una use on 26 Oc obe 2023
BigSeqKi | 9
Table 5: Execu ion imes (seconds) using di e en numbe o co es: eplace command. Highligh ed a e as es ime and numbe o
imes as e han sequen ial BigSeqKi
1 2 4 8 16 32 64 (2 nodes) 128 (4 nodes) 256 (8 nodes)
D
1
seqki 132.4 [1.02 ×] – – – – – – – –
BigSeqKi 134.5 69.5 36.1 25.0 18.7 12.7 [10.6 ×] 13.1 13.6 12.5 [10.8 ×]
D
2
seqki 395.7 [1.04 ×] – – – – – – – –
BigSeqKi 410.6 213.5 110.1 74.5 56.9 29.7 [13.8 ×] 16.8 13.9 13.5 [30.4 ×]
D
3
seqki 410.5 [0.99 ×] – – – – – – – –
BigSeqKi 406.7 209.5 109.4 74.0 56.1 29.5 [13.8 ×] 15.3 13.6 12.9 [31.5 ×]
D
4
seqki 543.7 [1.05 ×] – – – – – – – –
BigSeqKi 570.3 293.5 109.4 74.0 55.1 29.4 [19.4 ×] 20.3 13.5 12.5 [45.6 ×]
D
5
seqki 1,572.1 [1.03 ×] – – – – – – – –
BigSeqKi 1,621.7 819.9 420.1 217.2 115.1 62.9 [25.8 ×] 37.2 24.2 18.5 [87.7 ×]
D
6
seqki 8,980.8 [1.07 ×] – – – – – – – –
BigSeqKi 9,620.8 5,000.3 2,605.2 1,364.2 717.7 387.5 [24.8 ×] 142.1 90.5 60.2 [159.8 ×]
Table 6: Execu ion imes (seconds) using di e en numbe o co es: mdup command. Highligh ed a e as es ime and numbe o imes
as e han sequen ial BigSeqKi
1 2 4 8 16 32 64 (2 nodes) 128 (4 nodes) 256 (8 nodes)
D
1
seqki 178.9 [1.01 ×] – – – – – – – –
BigSeqKi 180.5 94.3 50.2 35.1 27.1 15.8 [11.4 ×] 14.8 14.4 13.8 [13.1 ×]
D
2
seqki 320.6 [1.04 ×] – – – – – – – –
BigSeqKi 333.3 174.7 93.5 65.9 49.9 26.5 [12.6 ×] 15.9 14.1 [23.6 ×] 15.0
D
3
seqki 515.5 [0.91 ×] – – – – – – – –
BigSeqKi 469.5 246.7 182.7 127.5 96.1 51.4 [9.1 ×] 27.4 20.9 20.6 [22.8 ×]
D
4
seqki 729.9 [0.99 ×] – – – – – – – –
BigSeqKi 720.5 378.5 197.5 139.7 102.9 54.0 [13.3 ×] 30.5 16.4 14.1 [51.1 ×]
D
5
seqki 2,173.6 [0.97 ×] – – – – – – – –
BigSeqKi 2,100.2 1,110.4 612.3 341.2 195.1 115.2 [18.2 ×] 70.5 43.2 28.1 [74.7 ×]
D
6
seqki 9,937.1 [1.11 ×] – – – – – – – –
BigSeqKi 11,022.3 5,578.5 3,006.7 1,709.6 1,004.1 600.1 [18.4 ×] 275.2 241.6 228.8 [48.2 ×]
Rega ding he expe imen al esul s, BigSeqKi clea ly ou pe -
o ms seqki , py as x , and sam ools o all he asks conside ed. On
a single se e , BigSeqKi is o e all ens o imes as e han hose
s a e-o - he-a ools, eac hing speedups wi h espec o he BigSe-
qKi sequen ial ime up o 27.7 ×. Conside ing an 8-node clus e ,
BigSeqKi is e en as e , eac hing speedups highe han 160 ×. I
means ha mos o he asks can be pe o med in jus a ew sec-
onds . Fo ins ance , ou oolki e ec i el y educes he execu ion
ime o he loca e command on ou la ges da ase om 3 days
o a me e 0.8 hou s. I is impo an o highligh ha seqki and
sam ools we e unable o p ocess ha da ase wi h some ou ines
due o memo y issues, which con i ms ha cu en s a e-o - he-
a ools a e no well i ed o p ocessing e y la ge iles.
As u u e w o k, w e plan o add also he emainde seqki com-
mands no included in he cu en e sion o BigSeqKi : sliding ,
sana , x2 ab , ab2 x , con e , amplicon , ish , spli , spli 2 ,
es a , and mu a e . No e ha all o hem a e independen ou-
ines, so hei implemen a ion using IgnisHPC will be s aigh o -
w a d.
A ailabili y o Sou ce Code and
Requi emen s
P ojec name: BigSeqKi
P ojec homepa ge: h ps://gi hub.com/ci iususc/BigSeqKi
Downloaded om h ps://academic.oup.com/gigascience/a icle/doi/10.1093/gigascience/giad062/7233988 by UDC - Uni e sidade da Co una use on 26 Oc obe 2023