Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
h ps://doi.o g/10.1186/s42400‑024‑00336‑3
RESEARCH Open Access
© The Au ho (s) 2025. Open Access This a icle is licensed unde a C ea i e Commons A ibu ion 4.0 In e na ional License, which
pe mi s use, sha ing, adap a ion, dis ibu ion and ep oduc ion in any medium o o ma , as long as you gi e app op ia e c edi o he
o iginal au ho (s) and he sou ce, p o ide a link o he C ea i e Commons licence, and indica e i changes we e made. The images o
o he hi d pa y ma e ial in his a icle a e included in he a icle’s C ea i e Commons licence, unless indica ed o he wise in a c edi line
o he ma e ial. I ma e ial is no included in he a icle’s C ea i e Commons licence and you in ended use is no pe mi ed by s a u o y
egula ion o exceeds he pe mi ed use, you will need o ob ain pe mission di ec ly om he copy igh holde . To iew a copy o his
licence, isi h p:// c ea i eco mmons. o g/ licen ses/ by/4. 0/.
Cybe secu i y
Building ala ge, ealis ic andlabeled HTTP
URI da ase o anomaly‑based in usion
de ec ion sys ems: Biblio‑US17
Jesús Díaz‑Ve dejo1 , Ra ael Es epa2 , An onio Es epa2* , Ja ie Muñoz‑Calle2 and
Ge mán Madinabei ia2
Abs ac
This pape in oduces Biblio‑US17, a labeled da ase collec ed o e 6 mon hs om he log iles o a popula public
websi e a he Uni e si y o Se ille. I con ains 47 million eco ds, each including he me hod, uni o m esou ce iden i‑
ie (URI) and associa ed esponse code and size o e e y eques ecei ed by he web se e . Reco ds ha e been
classi ied as ei he no mal o a ack using a comp ehensi e semi‑au oma ed p ocess, which in ol ed signa u e‑based
de ec ion, assis ed inspec ion o URIs ocabula y, and subs an ial expe manual supe ision. Unlike compa able
da ase s, his one o e s a genuine eal‑wo ld pe spec i e on he no mal ope a ion o an ac i e websi e, along wi h an
unbiased p opo ion o ac ual a acks (i.e., non‑syn he ic). This makes i ideal o e alua ing and compa ing anomaly‑
based app oaches in a ealis ic en i onmen . I s ex ensi e size and du a ion also make i aluable o add essing
challenges like da a shi and insu icien aining. This pape desc ibes he collec ion and labeling p ocesses, da ase
s uc u e, and mos ele an p ope ies. We also include an example o an applica ion o assessing he pe o mance
o a simple anomaly de ec o . Biblio‑US17, now a ailable o he scien i ic communi y, can also be used o model
he URIs used by cu en web se e s.
Keywo ds Anomaly de ec ion, In usion de ec ion sys ems, Da a acquisi ion, T aining da ase s, Web applica ion
il e s, Biblio‑US17 da ase
*Co espondence:
An onio Es epa
[email p o ec ed]
Full lis o au ho in o ma ion is a ailable a he end o he a icle
In oduc ion
Mo e han 200 million web se e s a e ope a ional on he
In e ne oday, acili a ing he deli e y o Web Se ices
and websi es. These se e s play a c i ical ole in ena-
bling use s o sha e in o ma ion and accomplish much
o hei daily asks (Fo bes. 2023; Ageed e al. 2021).
Consequen ly, he Hype ex T ans e P o ocol (HTTP)
now accoun s o a signi ican po ion o In e ne a ic
(Ib ahim e al. 2021). Howe e , his widesp ead use also
makes web se e s equen a ge s o cybe a acks and
malwa e dis ibu ion (Husák e al. 2021). Acco ding o
Apps (2021), some 30,000 websi es we e hacked daily in
2021. Mos o hese a acks a ge ulne abili ies in web-
si e so wa e, especially in con en managemen sys ems,
whe e plugins and ou da ed so wa e play a signi ican
ole in inc easing he isk o b eaches (BenMa in e al.
2022).
Web a ack de ec ion has adi ionally elied on
in usion de ec ion sys ems (IDS) (Aga wal and Hus-
sain 2018) and Web Applica ion Fi ewalls (WAFs)
(Pałka and Zacha a 2011). Many IDS and WAFs iden-
i y a acks by looking o ecognizable pa e ns in da a
o i s ea u es. Fea u e-based in usion de ec ion ech-
niques p ima ily ely on p ede ined in usion cha ac-
e is ics, known as Indica o s o Comp omise (IoCs), o
Page 2 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
iden i y known a ack pa e ns. As such, hey a e well-
sui ed o de ec ing known h ea s. Howe e , despi e
he a ailabili y o open-sou ce eposi o ies (Díaz-Ve -
dejo e al. 2022), IoC da ase s, wi h hei high accu acy,
a e o en conside ed aluable in ellec ual p ope y by
h ea in elligence endo s and a e ypically no made
public. On he o he hand, anomaly-based in usion
de ec ion sys ems (AIDS) ocus on iden i ying de ia-
ions om es ablished no mal beha io , making hem
pa icula ly e ec i e a de ec ing unknown o no el
a acks (which made up o e 42% o a acks in 2021
acco ding o some es ima es ze 2023). AIDS a e no
wi hou challenges, such as alse posi i es (Diaz-Ve -
dejo e al. 2024a) o hei eliance on quali y da ase s
o lea n no mal beha io (Be a e e al. 2018; Hajj e al.
2021; Kh aisa e al. 2019). Using aw da a, such as a -
ic and logs, is mo e app op ia e o anomaly-based
de ec ion because i cap u es he ull spec um o use
and sys em ac i i ies (Ga cia-Teodo o e al. 2009). The
da ase in oduced in his pape is designed speci ically
o anomaly-based IDS e alua ion, which is a eason-
able app oach gi en he na u e o he da a.
The challenges associa ed wi h cons uc ing eal-wo ld
log da ase s a e signi ican . Fi s ly, p i acy conce ns a ise
because hese da ase s o en con ain sensi i e in o -
ma ion, such as IP add esses, which o ganiza ions a e
eluc an o make publicly accessible o a oid p i acy io-
la ions. As a consequence, ep oducibili y issues eme ge
since e alua ions a e equen ly conduc ed on p i a e
da ase s, p e en ing he ep oducibili y o he esul s
and compa abili y o he p oposed app oaches. Secondly,
g ound u h de e mina ion becomes challenging amids
unknown use ac i i ies, as he labeling p ocess ideally
equi es expe examina ion, which is p ohibi i e in la ge
da ase s (Díaz-Ve dejo e al. 2020). On he o he hand,
syn he ic log da ase s gene a ed h ough simula ions
o e some ad an ages. They p o ide a mo e lexible ech-
nical en i onmen , enable he launch o a bi a y a acks,
and ensu e a eliable g ound u h due o he absence
o unknown ac i i ies (Gao e al. 2017). Howe e , he e
a e limi a ions o using syn he ic da a. Fi s ly, eal-wo ld
complexi y may no be ully cap u ed, as simula ions
migh miss e a ic use beha io o neglec ed con igu a-
ions, leading o a lowe a e o alse ala ms (Landaue
e al. 2022). Secondly, long- e m changes in sys em
in as uc u e o con en a e o en o e looked in sho -
e m simula ions, esul ing in a lack o ep esen a ion o
e ol ing en i onmen s. Las ly, he numbe and con en
o syn he ic a acks may no e lec he p opo ion, di e -
si y and sophis ica ion o eal-wo ld a acks. Gi en hese
challenges, he da ase in oduced in his pape aims o
b idge he gap by p o iding a ealis ic ye manageable
log da ase speci ically designed o anomaly-based IDS
e alua ion.
We p esen a new da ase in ended o scien i ic use in
he con ex o URI-based anomaly de ec ion: Biblio-US17
(b ie ly ou lined in Diaz-Ve dejo e al. (2024b)). Ou
da ase is de i ed om access log iles o he public web-
si e o he Uni e si y o Se ille Lib a y in Spain and has
h ee majo dis inc i e cha ac e is ics:
• I was collec ed in a eal ope a ing en i onmen .
• I con ains a la ge olume o eco ds (o e 47 million)
spanning a b oad pe iod (six mon hs), allowing o
he s udy o long- e m da a changes o some ex en .
• I is labeled: we conduc ed a ho ough semi-au o-
ma ed p ocess o de e mine g ound u h. This
equi ed a o midable e o , bu as a esul , a ealis-
ic model o no mal websi e beha io can be in e ed.
Addi ionally, he pe o mance o anomaly-based IDS
can be mo e accu a ely assessed using eal a acks
a ge ed a he ac i e websi e.
Biblio-US17 also has some limi a ions (see sec ion"Limi-
a ions and challenges"), he mos no able being ha
da a is es ic ed o he me hod, URI, esponse code,
and esponse size o each eques ecei ed by he se e .
While eco ds a e in ch onological o de , hey lack ull
imes amps, which limi s he use ulness o his da ase
o ime-based anomaly de ec ion echniques. Despi e
i s limi a ions, we a e con iden ha Biblio-US17 will
be a aluable addi ion o public da ase s, imp o ing he
e alua ion o u u e URI- ocused anomaly-based IDS and
enabling compa ison o di e en app oaches wi h ealis-
ic da a. We hope his e o will lead o b oade indus-
y ecogni ion and accep ance o academic esea ch
ou comes.
The emainde o his pape is o ganized as ollows.
Sec ion "Backg ound and ela ed wo k" discusses he
backg ound o c ea ing da ase s o web-a ack de ec ion
and e iews ela ed wo k. Sec ion"Acquisi ion and labe-
ling" desc ibes he p ocesses o da a acquisi ion, p epa a-
ion, and labeling. Sec ion"Da ase desc ip ion" p o ides
a de ailed desc ip ion o he da ase , including he s uc-
u e o iles, di ec o ies, and o ma . Sec ion"Sugges ed
use and illus a i e case s udy" add esses pa i ioning o
esea ch pu poses and includes an illus a i e case s udy.
Sec ion"Limi a ions and challenges" highligh s he main
limi a ions o he da ase , and sec ion"Conclusions" con-
cludes he pape .
Page 3 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
Backg ound and ela ed wo k
The c ea ion o sui able da ase s wi h eal-wo ld p ope -
ies is a challenging endea o (Sha a aldin e al. 2018).
Fo a da ase o be adequa e o AIDS de elopmen i
should a leas be (Viegas e al. 2017):
1. Rep esen a i e. This means ha he da a used should
ha e simila p ope ies o he da a ound in he sys-
em being p o ec ed, such as being alid, cu en , and
speci ic o he deploymen en i onmen . I is wo h
no ing ha using da a om ope a ional en i on-
men s is he mos e ec i e, whe eas syn he ic da a
may no accu a ely ep esen an ealis ic ope a ional
con ex (despi e ecen a emp s o easonably gene -
a e ealis ic log da a (Landaue e al. 2020; Ue z e al.
2021)).
2. Su icien ly la ge. The da ase should include enough
olume o da a o be pa i ioned and s ill be use ul
o aining, e alua ion o alida ion pu poses. The
aining da ase size should su ice o ain he no -
mali y model wi h s a is ical signi icance (Es epa
e al. 2020). Ideally, he collec ion pe iod should also
be long enough o add ess he issue o da a a iabil-
i y o e ime. Syn he ic da a gene a ed wi h sc ip s
usually lacks he a iabili y o da a om ope a ional
in as uc u es (Landaue e al. 2022).
3. Labeled. Each da a eco d should be classi ied as
no mal o an a ack. Howe e , labeling can be chal-
lenging when dealing wi h da ase s collec ed om
public ope a ional deploymen s, as hey may con ain
e o s o a acks (Mahoney and Chan 2003; B ugge
and Chow 2007). Fo his eason, da a sani iza ion is
ad ised (C e u e al. 2008) o iden i y a ack ins ances
in he collec ed da a and educe he numbe o unla-
beled a acks o da a e o s as much as possible. This
p ocess should ideally include manual expe supe -
ision (Paxson 2004), which is no easible o e y
la ge da ase s. A common me hod o educe he
sani iza ion cos is o u ilize a signa u e-based IDS o
iden i y he majo i y o known a acks (Díaz-Ve dejo
e al. 2022). S ill, expe supe ision is una oidable
i one aims o ob ain quali y da ase s ee o a acks
o undesi ed a i ac s (C e u e al. 2008). In con as ,
labeling is s aigh o wa d in a syn he ic da ase s
since all ac i i y is known.
Besides he abo e equi emen s, a da ase collec ed
om a public en i onmen should comply wi h appli-
cable p i acy egula ions, such as he Eu opean Union’s
Gene al Da a P o ec ion Regula ion (Hje ppe e al.
2019). Fu he mo e, secu i y should also be conside ed
as some da a could po en ially disclose in o ma ion
abou he web se e in as uc u e, acili a ing u u e
a acks. To add ess hese issues, p i acy-p ese ing
echniques such as gene aliza ion, supp ession o pe -
u ba ion should be applied (Fung e al. 2010). The la -
e consis o eplacing he o iginal da a alues wi h
syn he ic ones ha p ese e owne s’ p i acy while s ill
e ain use ul s a is ical in o ma ion (Rei e and Rubin
1998; Salaza -He nández and Díaz-Ve dejo 2010).
Da ase s sui ed o web a ack de ec ion
The URI ield in HTTP eques s plays a cen al ole in
iden i ying web a acks (Saxena e al. 2022). Indeed,
app oxima ely 85% o a ack signa u es used by com-
mon SIDS/WAF include he URI ield (Díaz-Ve dejo e al.
2022). This sec ion discusses da ase s cu en ly a ailable
o esea che s o de eloping and e alua ing anomaly-
based IDS ha speci ically a ge he con en o HTTP
messages, pa icula ly he URI. I is impo an o no e
ha we ha e excluded da ase s ha do no co e HTTP
messages, e en i hey ha e o he aluable p ope ies o
anomaly de ec ion, such as hose based solely on a ic
lows. I is also impo an o no e ha he e a e unla-
beled da ase s o HTTP messages collec ed om eal-
li e deploymen s, such as Schulz e al. (2021), Chodak
e al. (2020), ha a e a ailable. Howe e , hey ha e been
excluded om his sec ion because hey canno be used
o AIDS e alua ion, due o he absence o labeling.
To he bes o ou knowledge, Table1 lis s all pub-
licly a ailable labeled da ase s con aining he URI ield
o HTTP eques s. The able p o ides in o ma ion on
he sou ce ile o ma , labeling ca ego ies, whe he he
da ase was a i icially gene a ed o collec ed om an
ope a ional en i onmen , whe he labels we e manu-
ally supe ised, he collec ion da e (yea ) and pe iod,
and numbe o usable URIs in each da ase . A no iceable
common cha ac e is ic among hese da ase s is ha hey
ha e been a i icially gene a ed (i.e., syn he ic) and au o-
ma ically labeled a e gene a ion. The only excep ion is
NSL-KDD, whe e labels unde wen a subsequen supe -
ision. Only UNSW-NB15 con ains a mix o syn he ic
packe s (a acks) and li e-cap u ed ones. Howe e , he
labeling has been done wi hou supe ision.
F om he pe spec i e o da ase acquisi ion and con-
s uc ion, ou app oach signi ican ly di e s om hose
u ilized in he da ase s included in Table1. As p e iously
men ioned, o p io i ize e o - ee and ease labeling, all
da ase s in Table1 gene a e a ic syn he ically wi hin
con olled en i onmen s. A basic app oach is exempli ied
by DARPA99, whe e a ic is cap u ed du ing he simu-
la ed no mal ope a ion o use s in a con olled ne wo k.
In his case, all use -gene a ed a ic is assumed o be
no mal, while a ious ypes o a acks a e injec ed om
a speci ic s a ion in a con olled manne , acili a ing easy
Page 4 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
labeling. Con e sely, CIC-IDS2017 ep esen s a mo e
sophis ica ed app oach, whe e use p o iles a e es ab-
lished based on eal a ic moni o ing be o e syn he ic
a ic gene a ion. These p o iles aim o main ain p o-
po ionali y among di e en use ypes du ing da ase
gene a ion, he eby minimizing po en ial biases in he
gene a ed a ic. Howe e , e en in his app oach, a acks
a e syn he ically and con ollably in oduced, which does
no e lec hei ac ual equency in eal-wo ld scena ios.
In con as , ou app oach s a s om he opposi e si u-
a ion: we cap u e eal-wold a ic om an ac i e websi e,
including a i ac s, e o s, and po en ial a acks, and hen
p oceed o iden i y no mal and a ack a ic. Admi edly,
his app oach c ea es a “g ay a ea” whe e some eques s
canno be labeled due o a lack o in o ma ion o ambi-
gui ies. Howe e , simila si ua ions a e also obse ed in
some da ase s om Table1 (e.g., NSL-KDD). The main
ad an age o ou app oach is he ealism and ep esen a-
i eness o he da a, ee om biases associa ed wi h he
gene a ion o bo h clean a ic and a acks.
Rega ding da a olume, we obse e a signi ican di e -
ence in he numbe o eco ds o he da ase s in Table1
compa ed o ou s. A low size may nega i ely impac he
s a is ical signi icance o he no mali y models in e ed.
Fu he mo e, he use o simula ions esul s in educed
di e si y in he URIs, ypically esponding o ela i ely
simple se e s. Fo example, DARPA98 used wo se e s
wi h limi ed ocabula ies and a low numbe o a iables/
alues pe URI. In o he cases (e.g., CIC-IDS2017), a
ela i ely small numbe o URIs appea equen ly, lead-
ing o educed a iabili y. In no case is a con en man-
agemen sys em (CMS) was used a he se e (such as in
ou da ase ), and he complexi y o he se ice and URI
a iabili y is gene ally educed compa ed o ou s. Mo e-
o e , he equency and ype o a acks a e syn he ically
gene a ed in all da ase s, in oducing a signi ican bias in
de ec ion pe o mance assessmen s (TP/FP a es lacking
eal signi icance).
Due o he in a ian a ic gene a ion mechanism and
he cos o simula ion execu ion, he da ase s in Table1
lack su icien empo al du a ion and a iabili y o ana-
lyze issues ela ed o empo al d i . In con as , cap u -
ing a ic in a eal scena io is ela i ely s aigh o wa d,
allowing o ex ended collec ion pe iods, such as in ou
case. In ou app oach, he cos o building he da ase
is associa ed wi h he labeling p ocess a he han da a
gene a ion.
Finally, i is wo h no ing ha some o he da ase s
om Table1 a e e y old (e.g., om 2010 o olde ). As
a esul , he e is a isk ha he URIs (whe he malicious
o benign) may no accu a ely e lec cu en eques pa -
e ns, po en ially limi ing hei e ec i eness in e alua ing
AIDS p oposals in ended o ope a ional en i onmen s.
Biblio-US17, collec ed om a eal-wo ld ac i e web-
si e, is labeled wi h no mal and a ack classes. I ea u es
a la ge olume o eco ds, a collec ion pe iod sui able o
s udying ime shi issues o some ex en , and is ecen
Table 1 Public da ase s used in he li e a u e o web in usion de ec ion sys ems assessmen (Díaz‑Ve dejo e al. 2022)
* Classes: N (no mal), A (a acks), U (unlabeled), An (anomaly)
** Labels: G (As gene a ed), S (Supe ised)
Da ase Fo ma (URIs) Classes Real/Syn h Labels Only HTTP Yea Du a ion
#
URIS Commen s
DARPA’99 PCAP N/A/U S G N 1999 5w 100k Widely used o IDS esea ch, lawed, obso‑
le e, no h p speci ic a ic/a acks, only 2
web se e s
KDD’99 CSV N/A/U S G N 1999 5w – Pa ame e ized lows, di ec ly de i ed
om DARPA’99, inhe i DARPA’99 laws
NSL‑KDD PCAP N/A/U S GS N 2009 5w 10k Sani ized e sion o KDD’99, no o URI
based analysis
CSIC2010 LIST N/An S G Y 2010 – 96k Only URI, mixes a acks wi h anomalies,
single se e
UNSW‑NB15 PCAP N/A RS G N 2015 31h 27k Lack o de ails abou web se e s and a ic
ISCX‑URL2016 CSV N/A S G Y 2016 – – Ou going a ic, 114k URIs, no sui able
o ain a si e, highly unbalanced (only 35k
no mal URIs)
CIC‑IDS2017 PCAP N/A S G N 2017 5d 272k 392k URIs, mix o ou going and incoming
a ic, mos incoming URIS a e a acks, low
a iabili y in URIs (only 6k que ies and many
epea ed URIs)
Biblio‑US17 CSV N/A R S Y 2017 6m 47M HTTP log om public Web Se e wi h no ‑
mal and a acks URIs, high a iabili y in URIs,
wide imespan
Page 5 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
enough o e lec cu en web dynamics. The da ase also
bene i s om being old enough o some 0-day a acks o
now ha e public signa u es, wi h p i acy conce ns mi i-
ga ed as he se e is no longe ac i e.
Acquisi ion andlabeling
The da ase o igina es om he aces gene a ed by he
web se ice o he Lib a y o he Uni e si y o Se ille
(h p:// bib. us. es) be ween Janua y 1s and July 17 h,
2017. The web applica ion, de eloped wi h D upal 7.96
con en managemen sys em and he Apache web se e
( 2.2), p o ides in o ma ion abou bibliog aphic hold-
ings. As such, mos eques s a e bibliog aphic in o ma-
ion sea ches (e.g., book i les, pape s, hesis, au ho s)
and, o a lesse ex en , con en upda es om lib a y s a .
The websi e includes also news and se ice in o ma ion.
The si e, wi h o e 50,000 s uden s, expe iences high
demand and a wide ange o eques s, making i highly
dynamic.
Ou da a sou ce comes om he Apache access log
iles ha span 198 days and con ain a o al o 47,902,323
eques lines. Du ing his pe iod, he po al ansi ioned
be ween HTTP and HTTPS, so bo h se ice po s gene -
a ed log iles. Fo secu i y easons, he uni e si y did no
handed us he o iginal log da a iles bu a p ocessed copy
o hem (one daily ile named a e he da e o collec ion)
wi h he ollowing da a ields o each eques : me hod,
URI, p o ocol e sion, esponse code and esponse size
as shown in he ollowing box.
As illus a ed in Fig.1, we ca ied ou he asks o da a
p epa a ion and labeling wi h he iles ha we e gi en o
us:
(i) Da a p epa a ion, in which we p oduced a unique
ID o each eco d, supp essed unsui able eco ds
(e.g., mal o med messages acco ding o s anda d
syn ax o eques s wi hou he URI ield), and u -
he anonymized sensi i e in o ma ion.
(ii) Labeling, in which we conduc ed a semi-supe -
ised sani iza ion p ocess ha allowed us o de ec
a acks wi h di e en eliabili y le els (labeled as
LVL1 o LVL4) and u he il e unsui able eco ds
ha did no comply wi h s anda d syn ax o a i-
ous easons (labeled as OOS1 o OOS4).
We ob ained a inal coun o 47,402,996 sui able egis e s
om which he clean/a ack da ase s we e gene a ed. The
emainde o his sec ion elabo a es on hese p ocesses.
Da a p epa a ion p ocess
This p ocess aims o ob ain an ini ial, homogeneous,
well- o ma ed se o eco ds ha complies wi h p i acy
and secu i y egula ions and includes, a leas he me hod
and URI o each eco d.
P ep ocessing andcondi ioning
To c ea e a p ope ly-o ganized and well- o med da ase ,
we ca ied ou he ollowing s eps:
1. We elimina ed eques lines ha missed he URI
ield, which led o he emo al o 499,327 lines. Mos
o hese lines we e HEAD eques s wi hou a co e-
sponding esponse code.
2. Each eques was assigned a unique Regis e ID (RID)
label ha shows he collec ion da e and sequence
o de in he o iginal log ile. Fo example, he RID o
he 123 d eco d in he HTTPS log ile da ed Ma ch
1s would be [03-01-S000123]. The RID can also
be seen as a basic empo al e e ence, indica ing he
da e and o de o he eques a he han a ull imes-
amp.
3. We u ilized a pa se o check he con o mi y o
eques s wi h s anda d syn ax. I possible, he issues
iden i ied (e.g., cha ac e ’
’ ending he eques , un-
ca ion e o s, e c.) we e ixed. O he wise, mal o med
eques s we e elimina ed, esul ing in he emo al o
89 egis e s.
Fig. 1 Sani izacion s eps as de i ed om Díaz‑Ve dejo e al. (2020)
Page 6 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
By he end o his ask, he da ase con ained a o al o
47,402,907 eques lines, each wi h a unique RID and he
ields shown in he example abo e. These lines unde -
wen he anonymiza ion p ocess desc ibed nex .
Anonymiza ion
This p ocess aims o de ec and ob usca e sensi i e
in o ma ion ha could comp omise he websi e secu-
i y by disclosing se ice in as uc u e, o iola e da a
p o ec ion egula ions. This s ep was compulso y and
equi ed he coope a ion and app o al o independen
and ex e nal pe sonnel (legal and secu i y eams). To
achie e his, we ca ied ou wo sequen ial asks:
1. Iden i ica ion o Sensi i e In o ma ion: we sea ched
o in o ma ion ela ed o pe sonal da a o se e
secu i y (conside ing pe cen encoding) in he URIs.
To iden i y Pe sonal Da a, we ca e ully sea ched o
speci ic keywo ds and pa e ns, such as login c eden-
ials, passwo ds, names, emails, o use IDs. Al hough
he sea ch was au oma ed, e e y occu ence was
manually inspec ed. As a esul , his p ocess was
ime-consuming and ook abou wo mon hs o com-
ple e. We ound ha mos o he sensi i e in o ma-
ion belonged o he au ho s o he pape s o books
use s we e sea ching o , no he se ice use s. How-
e e , we also iden i ied some sensi i e pe sonal da a,
including 62 di e en Uni e si y o Se ille use IDs
(u us), wo IP add esses, and one email add ess. To
iden i y In as uc u e Sensi i e Da a, we sea ched
o ins i u ional se e names (i.e., domain name us.
es) and ilenames wi h po en ial sensi i e in o ma-
ion. A o al o 4371 di e en ilenames (wi h ex en-
sions such as.doc,.js,.pd ,.pps,.pp ,.sw ,. x and.xls)
we e ound in mo e han 16 million URIs. Rega ding
ins i u ional se e s, we ound 27,187,785 URIs ha
con ained up o 44 di e en se e names. All he
in o ma ion iden i ied in his ask (e.g., ilenames,
se e names, use ’s sensi i e in o ma ion) accoun ed
o 4480 ex s ings, which should be ob usca ed in
he co esponding URIs o p e en u u e a acks on
he in as uc u e and o comply wi h p i acy egula-
ions.
2. Pseudo-anonymiza ion o sensi i e in o ma ion: we
applied a pseudo-anonymiza ion p ocedu e o he
s ings iden i ied in he p e ious ask. To p ese e o
a la ge ex en URI cha ac e is ics, we used a subs i-
u ion echnique wi h he ollowing cha ac e is ics
(Salaza -He nández and Díaz-Ve dejo 2010):
• Dic iona y-based: e e y occu ence o he same
o iginal s ing will unde go he same subs i u ion.
• The subs i u ing s ing keeps he ollowing s a-
is ical p ope ies o he o iginal s ing: leng h,
numbe o alphanume ic cha ac e s, and numbe
o uppe and lowe cha ac e s. Special cha ac e s
(no alphanume ic) a e p ese ed and kep in hei
o iginal posi ion since hey a e commonly pa o
malicious URIs (Saxena e al. 2022).
• Filename ex ensions a e p ese ed.
We s a ed he subs i u ion p ocess by he longes s ing
o a oid i e a i e subs i u ions. Table2 shows some illus-
a i e examples o subs i u ions (in bold).
While ob usca ion a ec s he da a used o lea n-
ing o e alua ing AIDS, i s impac a ies depending on
he ex en and me hod o s ing subs i u ion. We aimed
o minimize his impac on he da ase . Ul ima ely, we
subs i u ed 5.4% o he URIs’ s ings, while p ese ing
mos o he URI cha ac e is ics ele an o classi ica ion
(Saxena e al. 2022; Abad e al. 2023). We conduc ed a
p elimina y assessmen o he impac o he anonymiza-
ion p ocess based on known a ack de ec ion. The non-
anonymized a ack da ase would ha e only 3 addi ional
a acks (ou o +30k), while he clean da ase emains
unchanged.
The da ase ob ained a e he anonymiza ion p ocess
is published unde he name ull da ase (see Fig.1) and
con ains all he egis e s conside ed in he labeling phase.
Table 2 Examples o anonymiza ion
O iginal URI Pseudonymized URI Sensi i e in o ma ion
/a qui ec u a/use s/adol o /a qui ec u a/use s/kx acj u us (use id)
/sea ch/google/adol o /sea ch/google/kx acj
/sea ch/node/beni o2 /sea ch/node/hy3u a
/si es/se e 1.co.es/ iles/Jhon_1234_webSocke .pd /si es/xzy2kpq.kp.le/ iles/nHom_3782_pN dp bq .pd Domain and Filename
/ u ismo/si es/se e 1.co.es/ iles/Viajes20-22.doc / u ismo/si es/xzy2kpq.kp.le/ iles/h Y1gbn7-98.doc
/si es/se e 2.co.es/ iles/ /si es/kjpqb7g.bn.y / iles/
/si es/se e .us.es/ iles/js/js_ p4-pa9.js /si es/pomnh .mn.d / iles/js/js_p8m-n1 .js
Page 7 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
Labeling p ocess
We conduc ed a ho ough s udy o he ull da ase o
es ablish g ound u h and label each eco d as clean,
a ack, o no in compliance wi h s anda d syn ax. The
p ocess is ou lined in Fig.1, and, in gene al e ms, i ol-
lows he sani iza ion me hodology p oposed in Díaz-
Ve dejo e al. (2020). We ha e ollowed h ee sequen ial
s eps: (S1) use o signa u e-based IDSs o de ec known
a acks, (S2) manual supe ision o he ale s p oduced
du ing S1 o de ec alse posi i es and con i m ue posi-
i es, and (S3) inspec ion o he URI ocabula y: a semi-
au oma ed ask o ind new a acks o non-complian
egis e s. The ollowing subsec ions elabo a e on hese
s eps.
Known a acks de ec ion (S1)
We used he signa u e-based IDS ool Inspec o Log
(Díaz-Ve dejo e al. 2024c) (a ool o o line p ocessing),
o de ec a acks in log iles by applying he de ec ion
ulese s om he o icial eposi o ies o he open-sou ce
IDSs Sno (Tal 2023; ETo 2023), Nemesida ( ee e sion)
(Nem 2023), and he WAF ModSecu i y (OWA 2023).
Only ules including he URI ield we e selec ed excep
o hose labeled as DELETED in he Talos and ETOpen
eposi o ies. ModSecu i y was un using pa anoia le els
1, 2 (le el 3 was disca ded due o an o e whelming a e o
alse posi i es). Table3 shows, o each ulese , he num-
be o ules selec ed, he numbe o ale s p oduced, and
he numbe o di e en URIs e e enced in hese ale s.
The esul s in columns wo o ou show ha Sno ’s ules
p oduced mos o he ala ms (83% o hem). A o al o
283,165 URIs (ba ely 0.6% o he URIs in he ull da ase )
we e classi ied as a ack by hese signa u e-based IDSs.
Inspec ion (S2)
Each URI classi ied as an a ack by Inspec o Log was man-
ually inspec ed by an expe o de e mine whe he i was
a ue posi i e (TP) o alse posi i e (FP).1 URIs ound o
be TPs we e u he classi ied and labeled acco ding o
hei soundness and scope as ei he indubi able (labeled
as LVL1) o con ex -dependen (labeled as LVL2). The
la e indica es ha , al hough he URI is conside ed an
a ack in ou pa icula con ex , i could po en ially be
non-malicious in a di e en ope a ing en i onmen . The
igh mos pa o Table3 (columns i e o se en) shows
he esul s ob ained o each ulese . The las column
shows he a ack le el ca ego y (i.e., LVL1 o LVL2) o he
URIs manually e i ied as TP. In he bo om ow, Table3
shows he o e all numbe o a acks: 273,943 (
≥
99%
indubi able), which accoun s o 0.58% o he egis e s.
I is wo h no ing ha , du ing manual examina ion,
we ound ha mos o he a acks de ec ed by Talos and
ETopen ules we e ela ed o a pe cen encoding a ack
using he sequence %252525... wi h inc easing size
( he e we e 314,076 URIs using %252525). Mo e spe-
ci ically, in 257,258 cases, he URI size was g ea e han
1450 cha ac e s (which is a h eshold de ined in he ule
SID=17,410). Mo eo e , only 15,076 o he known a acks
de ec ed in his phase we e no ela ed o his s ing,
which would lead o an a ack a e o 0.03 % o he URIs
in he ull da ase .
Vocabula y inspec ion (S3): segmen a ion‑based analysis
This s ep aims o u he disco e un ecognized a acks
o URIs ou o speci ica ion in o de o build a quali y
da ase . Fo his, we ollowed he semi-supe ised me h-
odology desc ibed in Díaz-Ve dejo e al. (2020) in which
he e ision ocuses on he di e en ex s ings ha
compose he URI (some 80k) a he han he URIs (some
47M). The me hod can be summa ized as ollows:
• Dic iona ies o ma ion. URIs deemed clean (which
also include FP in S2) a e pa sed and segmen ed o
Table 3 A ack de ec ion esul s pe ulese
Rulese #Rules # URIs classi ied as
Ac i a ed #Ale s #URIs #FP #TP (LVL1/LVL2)
Talos (22/02) + ETopen (22/03)(Sno ) 175 287,253 278,404 8571 269,833 (26,8148/1685)
Nemesida (21/11) 125 34,520 19,252 627 18,625 (17,774/851 )
CRS3.3.2‑PL1 (ModSecu i y) (22/04) 43 18,345 9206 108 9098 (9098/0)
CRS3.3.2‑PL2 (ModSecu i y) 75 22,629 10,085 243 9842 (9482/0)
To al 418 344,942 283,165 9222 273,943 (272,103/1840 )
1 A p elimina y ule classi ica ion helped educe he amoun o manual
wo k by deac i a ing hose ules ha gene a ed exclusi ely alse posi i es
(e.g., SIDs 41742 and 1852 om he TALOS eposi o y, which e e o
obo s and gene a ed mo e han 50,000 FPs).
Page 8 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
gene a e h ee dic iona ies om he ex s ings
(i.e., wo ds) ound be ween s anda d delimi e s ha
cons i u e he pa h, key y alue pa s o he URI. To
build he dic iona ies, we used he U ipa se 2 lib a y,
which is RFC 3968 complian . Du ing he segmen a-
ion p ocess, he pa se iden i ied 169 non-complian
URIs due o he p esence o cha ac e s no pe mi ed
o an inco ec sequence o ields. The egis e s con-
aining hese URIs we e labeled as Ou -o -Speci ica-
ion (OOS) and igno ed in pos e io analysis. Table4
shows he numbe o di e en wo ds lea ned in each
dic iona y (size) and numbe o ins ances o such
wo ds in he URIs ha we e subjec o ocabula y
inspec ion (a e S2).
• Vocabula y inspec ion: The ocabula ies in hese dic-
iona ies we e manually supe ised seeking aces o
a acks. This equi ed a o midable amoun o wo k.
URIs con aining wo ds associa ed wi h a acks we e
labeled acco dingly wi h a ce ain le el o con i-
dence. A p elimina y e iew o he h ee dic iona ies
allowed us o iden i y some issues in he segmen a-
ion ca ied ou in he pa h and key dic iona ies,
which led us o u he classi y some URIs as OOS.
The las wo columns o Table4 show he numbe o
di e en wo ds lea ned in each dic iona y (size) and
numbe o ins ances o such wo ds in he URIs a e
excluding he URIs labeled as a ack o OOS in S3.
The main issues ound in each dic iona y a e summa-
ized bellow ( he in e es ed eade is e e ed o A.1
o u he de ails).
– Key Dic iona y. We iden i ied common pa e ns ol-
lowed by 1579 wo ds (86%) o his dic iona y, which
we e ound o be no mal a e inspec ion (seeA.2).
The emaining wo ds we e inspec ed in sea ch o
po en ially oublesome cha ac e s acco ding o
RFC 3986 such as gen-delims: (: / ? # [ ] @)
o sub-delims (! $ & ’ ( ) * +,; =). As
a esul o his e ision, some URIs we e labeled as
OOS ( u he de ails a e inA.2).
– Pa h Dic iona y. As wi h he p e ious dic iona y,
he sea ch o delimi e s and a p elimina y inspec-
ion o he pa h- ela ed ocabula y e ealed some
encoding issues (labeled as OOS) and seman ic
e o s (see A.3 o u he de ails). We disco e ed
e idence o some a acks based on he use o inco -
ec o malicious encoding. The egis e s con aining
hese malicious wo ds we e labeled as a ack LVL3.
Du ing he examina ion o his leng hy dic iona y,
we iden i ied bu s s o eques s wi h an ob ious
in en o scanning. We labeled hese eques s as
a ack LVL2. Simila ly, we also iden i ied one bu s
wi h iden ical eques s (i.e., e e ing o he same
esou ce), whose URLs we e labeled as a ack wi h
a new con idence le el LVL4. The iden i ica ion o
hese malicious eques s was challenging because
male olen en ies we e in e lea ed wi h no -
mal egis e s, and because each indi idual eques
seemed appa en ly non-malicious. As such, LVL4
a acks a e he leas sound ca ego y. We also iden i-
ied ha bu s s associa ed wi h scans p oduced p e-
dominan ly 300- ype esponses. Due o he impos-
sibili y o de ec ing all bu s s in he en ies and
he ac ha he majo i y o hese scans p oduced
a 300- ype esponse, we inally op ed o including
only egis e s wi h esponse code
<300
in he clean
da ase .
– Value Dic iona y. A pa e n analysis allowed us o
iden i y h ee legi ima e p e alen pa e ns (some-
imes wi h hexadecimal cha ac e s). A delimi e -
based sea ch allowed us o iden i y app oxima ely
1000 (mos ly legi ima e) wo ds. We inally classi ied
as LVL2 a ack hose URIs ha con ained wo ds
wi h he cha ac e ’ a he end o in combina ion
wi h newline. Finally, gi en he modes numbe
o anomalies disco e ed in he wo ds o his dic-
iona y, we analyzed he combina ions key= alue
ex ac ed om he que y pa s o he URI (a o al
o 20,039 unique combina ions). The esul s o his
inspec ion allowed us o iden i y h ee combina-
ions as OSS le els 2 and 4, and h ee key- alue
pai s as OOS2 (one wi h 1037 ins ances).
The main indings o he ocabula y inspec ion (S3)
can be summa ized as:
• se e al URIs did no comply wi h s anda d syn ax
o made an inadequa e use o pe cen encoding.
A ec ed egis e s we e classi ied as OOS1 and OOS2
espec i ely.
• se e al pa h segmen s included a acks based on he
mul iple encoding o he cha ac e ’%’ (%252525...) o
Table 4 Dic iona ies o med om he da ase
Segmen Ini ial (A e S2) Final (A e S3)
Size # Wo ds Size # Wo ds
pa h 51,130 275,392,782 26,488 259,515,518
key 1971 6,823,291 1578 6,689,066
alue 28,513 4,732,008 27,559 4,637,691
2 h ps:// gi hub. com/ u ipa se / u ipa se .
Page 9 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
malicious encoding. A ec ed en ies we e classi ied
as a ack (LVL2 o LVL3).
• de ec ion o bu s s p esumably associa ed wi h DoS
and scanning a acks (LVL2 o LVL4).
• De ec ion o a signi ican numbe (mo e han 55k) o
new indubi able a acks unno iced by he signa u e-
based de ec ion om S1 (LVL1).
Table5 summa izes he esul o his phase. We inally
es ablished ou ca ego ies o a acks (LVL1 o LVL4),
and ou ca ego ies o OOS (OOS1 o OOS4) o ge a
ine -g ain labeling o he URIs ha exhibi issues in hei
syn ax o ep esen a ion. Mo eo e , ou ocabula y anal-
ysis has ad ised o supp ess some en ies based on hei
associa ed esponse code in o de o a ain a mo e qual-
i y clean da ase , since a signi ican pa o he egis e s
associa ed o ResponseCode (RC) 300 we e deemed as
ei he a ack o OOS.
I is wo h no ing ha s age S3 added 74,422 new
a acks o hose iden i ied a e S23 (an addi ional 27%)
which highligh s he impo ance o his phase in he sani-
iza ion p ocess.
Da ase desc ip ion
Following he sani iza ion and labeling p ocess desc ibed
abo e, his sec ion de ails he iles and s uc u e o Bib-
lio-US17, which can be downloaded om Díaz Ve dejo
e al. (2023). A b ie summa y o his sec ion is also a ail-
able a Diaz-Ve dejo e al. (2024b).
Files and o ma ing
The da ase comp ises se e al ex iles o ganized in
di ec o ies. Each line in hese iles e e s o a eco d
h ough a eco d iden i ie RID which ela es i o i s
sou ce log ile and can be aken as a empo al e e ence
o he eco d, indica ing he da e and o de o collec ion.
The RID syn ax is as ollows:
‘[’MM-DD-Fnnnnnn’]’
whe e MM-DD s ands o he mon h and day o he
o iginal log ile, F s ands o he web se ice ype (A o
HTTP o S o HTTPS), and nnnnnn is an in ege ha
indica es he o de o he eques in he o iginal log ile.
An example RID would be [02-18-A001234], which
would co espond o he eques line 1234 om he o ig-
inal ile o he HTTP se ice da ed on Feb ua y 18.
Files a e a anged in di ec o ies and mos ile-
names ha e a common o ma : biblio-2017-mm-
dd.
<ex >
, whe e mm and dd e e o he mon h and
day o he o iginal log ile, and he ex ension
<ex >
de e mines he di ec o y and con en o ma as shown in
Table6.
Table 5 Numbe o URIs labeled du ing ocabula y analysis (S3)
A ack (74,422 egis e s) Ou o speci ica ion (10,352 egis e s)
Label Desc ip ion # Regis e s Label Desc ip ion # Regis e s
LVL1 Undubi uous 55,803 OOS1 Ou o speci ica ion (RFC3986) 169
LVL2 Con ex dependen /scanning 8794 OOS2 Ex ended cha s codi ica ion e o s/no
allowed cha s 2021
LVL3 Codi ica ion a ack 5515 OOS3 Ini ial ’//’ 6567
LVL4 Bu s a ack/DoS 4310 OOS4 Seman ic e o s/o he s 1595
Table 6 Di ec o ies, da a o ma and ile ex ensions o Biblio‑US17
Di Fm Ex Con en
RAW Raw . aw Full da ase : All alid egis e s a e anonimiza ion
LABELS Label .lbl Labels assigned du ing sani iza ion, indexed by [RID]
SID SID .sid Ale s gene a ed by signa u e IDSs, indexed by [RID]
CLEAN Raw .cl Regis e s conside ed clean indexed by [RID]
ATTACK Raw .a Regis e s conside ed a ack (only LVL1).
PARTITIONS Raw . ain . es . al Sugges ed pa i ions o clean eques s o use in ain/
es / alida ion.
AUX Va ious – Auxilia iles ( ules, sc ip s, ools)
3 Mos o hem—55,498—co espond o he use o ’%252525.... Excluding
his a ack, S3 added 15,076 new a acks.
Page 16 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
some imes led o he e ision o he URIs whe e he
wo d is ound) allowed us o inden i y he ollowing:
• a o al o 36 LVL2 a acks,
• 25 URIs did no comply wi h he speci ica ion and
we e labeled as OOS2 as he lack o compliance was
due o encoding issues
• 28 URIs seman ically w ong. These URIs had he
s ing *QUERYSTRINGPII_REMOVED* as key,
p esumably due o some kind o il e ing applied by
he se e o e he aces. As a esul , we added a
new ca ego y o he OOS label, OOS4 which was
used o ma k hese URIs.
I is wo h no hing ha he analysis o he keys x1 and
login allowed us o de ec a acks. In bo h cases, hey
a e con ained in bu s s o eques s ecei ed in June 1s
and June 6 h.
Fu he de ails
Ou sea ch o p oblema ic cha ac e s only p oduced
posi i e esul s o ’;’ and ’=’ in h ee si ua ions:
• The key amp;amp;amp;amp;amp;opción,
which ended up being labeled as OOS2.
• q=node/add was encoded as Q%3Dnode/add,
which was labeled as OOS4.
• 24 URIs did no encoded he cha ac e ’=’ a he
end o he alue ield associa ed wi h a couple o
suspicious keys. We labeled hem as OOS2.
Sea ching key wo ds ha e allowed as o de ec some
a acks o ype 2 in he ollowing cases:
• Que y ’login=cmd’. I is ound epea edly wi h
small a ia ions in he pa h add essed o speci ic
loca ions used by wo dp ess, which led us o classi y
i as a scanning.
• URIS p esumably also associa ed wi h wo dp ess
using he o ma /si es/wdn4.mu.xs/ iles//
wp-admin/[PAGE].php?x1
Pa h dic iona y
We ca ied ou se e al analysis gea ed owa d he de ec-
ion o di e en ypes o anomalies in he ocabula ies.
Fi s , and as a esul o he indings du ing S2, we sup-
p essed om subsequen analysis 18,394 s ings ha con-
ained %252525. The URIs ha con ained hese s ings
we e di ec ly labeled as LVL2 a ack.
Sys ema ic analysis: delimi e s
We sea ched and ound 101 wo ds wi h delimi e cha -
ac e s, which a e inspec ed manually. O hese, 2 a e
associa ed wi h LVL1 a acks, 29 wi h coding p oblems
(OOS2) and 49 wi h seman ic e o s (OOS4).
Use o pe cen encoding
We de ec ed issues ela ed wi h he inco ec use
o pe cen encoding. These issues we e ela ed o an
Fig. 4 Tempo al e olu ion o he ocabula ies lea ned
Table 10 Vocabula y de ails o (p esumably) ile names by
ex ension (a e phase 1)
Ex . #Wo ds #Ocu ences
.png 2334 20,548 545
.css 1710 6,529,799
.pd 1704 332,695
.jpg 1467 2,556,541
.h ml 1452 11,979
.js 1324 8 983,781
.asp(?) 353 2371
.php(?) 321 18,005
.h m 254 2311
.xml 126 749,907
.gi 123 1,031,787
.pp (?) 134 7918
.xls(?) 111 5141
.zip 90 1158
.sw 70 1161
.pps(?) 60 3167
o he s 381 27,279,872
To al 12,041 172,619,409
Page 17 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
ex ensi e use o encoded cha ac e s inside he wo ds
lea ned gi ing ise o excessi e wo d sizes o e one-
ous o impossible encoding (e.g., %EF%BF). Thus, we
ex ac ed and analyzed all wo ds ha included pe cen
encoding (10,738 wo ds). To acili a e manual inspec-
ion, we ca ied ou an i e a i e p ocedu e ha con-
sis s o subs i u ing (o ma king i applies) he di e en
encoding ound s a ing by he longes ones (encoding o
a cha ac e using UTF wi h 3 elemen s) un il he e is no
mo e wo ds wi h encoding cha ac e s. On each case, we
classi y he subs i u ed cha ac e as ei he as (associa ed
wi h) a ack, ou o speci ica ion, o admi ed. In he wo
i s cases, we il e ed he wo d and he co esponding
URIs om he clean da ase . The manual inspec ion o
some o hese URI e idences a acks based on he inco -
ec o malicious use o encoding. As such, a e labeled
as LVL3 such as we did wi h he a ack %2525.... An
example o one o hese egis e would be he ollowing:
On he o he hand, some encoded ex gi es ise o cha -
ac e s ha , al hough co ec om he he speci ica ion
poin o iew, a e seman ically inadequa e ega ding he
con en o he web applica ion. Fo example:
whe e %c3ndice is decoded as Ãndice a he han
índice, which does no make sense.
As a esul , we iden i ied 43 encodings o be excluded
(e.g., %EF%81%BD, %C2%A0) and 5 (%3A, %3E,
%7C, %3C and %ED) equi ed addi ional supe ision.
F om hose o be excluded, 22 a e associa ed wi h a acks
LVL3 and 21 cha ac e s seman ically inadequa e (OOS4).
The encodings equi ing u he supe ision ha e 34 di -
e en wo ds associa ed. Mos o hem, 32 wo ds, ended
[01-01-A016709] GET
/comunicacion/no icias/ca %C3%83%C6%92%C3%82%C2%A1l
ogo-
ama? page=2 HTTP/1.1"
200 22376
[01-03-A065330] GET /a qui ec u a/no icias/%c3ndice-jc -2014 HTTP/1.0" 200
135982
up labeled as OOS4 a e inspec ion. In o al, we iden i-
ied 5518 URIs o ype LVL3 and 396 o ype OOS4.
Sea ch o pa e ns
We iden i ied wo pa e ns ollowed by se e al wo ds:
• ’css_([a-Z]|_|-|[0-9]){40,}.css’,
ound in 1612 wo ds, and
• ’js_([a-Z]|_|-|[0-9]){40,}.js’, ound
in 1238 wo ds.
Bo h pa e ns we e conside ed no mal a e he inspec-
ion o a signi ican numbe o wo ds.
Filenames by ex ension
We ex ac ed he ex s ings ha p eceded ce ain ile
ex ensions. We iden i ied 16 ile ex ensions wi h mo e
han 50 occu ences (see Table10) ha added up a o al
o 8745 di e en wo ds. The emaining ex ensions added
273 di e en wo ds. We inspec ed hese wo ds g ouped
by ex ension seeking anomalies o pa e ns. We did no
ind anomalies o wo ds ha equi ed u he in es i-
ga ion, bu in he case o he ex ensions .php, .zip y
.asp, we did in es iga ed in u he de ail.
In he examina ion o he wo ds and URIs ha co e-
spond o hese ex ensions, we ound some eques s ha
could in end o scan since hey cons i u e a sequence o
eques s ha access o known loca ions om o he po -
als o ools h ough a ia ions in he pa h aimed o con-
i m he exis ence o a esou ce. In some cases, hey seem
o be gea ed owa d ins alling a ool o add-on. A e
con i ming wi h he adminis a o s ha hese eques s
Page 18 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
did no come om hem, and due o he di icul y o de e mining all he bu s in ended o scan, we decided o label
hese bu s o eques s as LVL2.
We also iden i ied bu s s o iden ical ( alid) eques s, which could be aced back o DoS a acks. In his case, we c e-
a ed a new ca ego y, LVL4, o label hese sequence o iden ical eques s. Like in he case o scanning eques s, i was
ha d o us o iden i y all hese bu s s because hey we e equen ly in e lea ed wi h no mal a ic.
Addi ional De ails
Fig. 5 Examples o LVL1 a ack
Fig. 6 Examples o LVL1 a ack
Page 19 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
We ha e iden i ied 835 eques s wi h que ies ha con ained z3=[al anume ic cha s]%3d%3d and
z4=z4=Lw%3d%3d ha a e included in bu s s and a e appa en ly associa ed o so wa e upda es.
Appendix B: Examples o a acks andOOS URIs
Examples o LVL1 a ack iden i ied du ingS2
See Fig.5.
Examples o a acks manually iden i ied du ingS3 (LVL1)
See Fig.6.
Examples o LVL2 a acks manually iden i ied du ingS3
See Fig.7.
Example o abu s a ack
See Fig.8.
Appendix C: Main challenges aced
Below, we ou line some o he signi ican challenges we
encoun e ed and how we add essed hem, p o iding
p ac ical guidance o he esea ch communi y in apply-
ing ou me hodology o c ea e new da ase s om eal-
wo ld da a. Some o hese aspec s ha e been discussed in
mo e de ail h oughou his a icle in he co esponding
sec ions o appendices.
• Da a Volume: The la ge numbe o ex eco ds and
successi e p ocessing phases necessi a ed he use o a
ela ional da abase. To uniquely iden i y eco ds and
acili a e indexing, each collec ed URI was assigned a
unique iden i ie . P ocessing was conduc ed acco d-
ing o labels associa ed wi h a ious p ope ies (e.g.,
de ec ion esul s om S-IDS), using di e en da a-
base ables.
Fig. 7 Examples o LVL2 a ack
Fig. 8 Example o a bu s a ack
Page 20 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
• De ec ion o Known A acks: Two majo challenges
a ose: some signa u e-based IDS equi e ne wo k
aces as inpu , necessi a ing eques gene a ion
and cap u e, which is compu a ionally expensi e
and p one o a i ac s. Addi ionally, some ools lack
co ela ion be ween ou pu s and inpu s, complica -
ing he iden i ica ion o ale - igge ing eco ds. We
add essed hese by de eloping Inspec o Log, a ool
ha p ocesses log iles and implemen s de ec ion
echniques om online ools, ensu ing clea mapping
be ween ou pu s and inpu eco ds ia unique iden i-
ie s.
• P ep ocessing and Condi ioning: Al hough da a
was p o ided by he ope a o om hei log iles,
i equi ed mino co ec ions due o e o s such as
incomple e lines, imp ope ield delimi a ions, and
inapp op ia e URI alues. The da abase and unique
iden i ie s we e c ucial o e iewing a ec ed eco ds
wi hou ep ocessing all da a.
• Anonymiza ion: P i acy conce ns equi ed ex en-
si e anonymiza ion, guided by an ex e nal audi .
We implemen ed an algo i hm ha modi ies sensi-
i e da a while p ese ing ea u es use ul o de ec-
ion (e.g., special cha ac e s, s ing equencies). The
p ocess was in ensi e, equi ing mul iple i e a ions
and independen e i ica ion be o e app o al by he
Uni e si y’s legal and secu i y eams. E alua ing he
impac o anonymiza ion on he esul s is also neces-
sa y.
• Realism: We minimized modi ica ions o p ese e
he ealism o he da a. All alid eco ds a e p o-
ided in he aw da ase , ega dless o ca ego iza ion.
Despi e no maliza ion by Apache p ep ocesso s,
some acqui ed da a lacked expec ed cha ac e is ics.
We p ese ed URIs as s o ed in he ace ile, e lec -
ing he ac ual se e -p ocessed eques s. Reco ds
wi h esponse codes ou side he 200 ange we e
excluded om he clean da ase bu e ained in he
aw da ase .
• Re ision o Ale s: TP/FP Classi ica ion (Phase S2 o
Labeling): We iden i ied signa u e-based IDS ules
gene a ing TP o FP and analyzed all de ec ed URIs.
SID-based analysis acili a ed his p ocess by e eal-
ing he de ec ion a ionale.
• Manual Supe ision (Phase S3): The mos challeng-
ing ask was he manual supe ision o he ocabu-
la y and associa ed URIs. Supe ising +47 M o
indi idual URIs was impossible, bu o e seeing hei
cons i uen s ings (a ound 80k) was easible. We
employed a ious ools, u ili ies, and Py hon sc ip s
o assis wi h he analysis, sea ch, and compa ison o
s ings. His og ams and wo d/pa e n lis s, such as
a ack-associa ed s ings (e.g., /e c/passwo d)
and alid s ings (e.g., 32 consecu i e hexadeci-
mal cha ac e s ollowed by .css), we e gene a ed
o expedi e he p ocess. In some cases, OSINT was
needed o de e mine he na u e o ce ain URIs. The
e o paid o , as i signi ican ly impac ed he clean
da ase by de ec ing mo e han 74k new a acks wi h
a ious con idence le els.
In summa y, his wo k was a challenging ask equi ing
signi ican expe ise and esou ces.
Acknowledgemen s
We would like o hank he lib a y o he Uni e si y o Se illa and, pa icula ly,
o Claudio A jona, and p o esso s Ra ael Ma inez Gasca and Angel Luis Va ela
o hei suppo in his wo k, e iewing he anonymiza ion p ocess and help‑
ing o achie e adminis a i e pe missions. We also ex end ou g a i ude o he
anonymous e iewe s. Thei eedback has been impo an in enhancing his
pape .
Au ho Con ibu ions
JD‑V has conduc ed he esea ch and has la gely aken he wo k o sani iza‑
ion and o ganiza ion o he da ase . He also has e iewed and co ec ed he
pape and w i en he i s d a . RE has co‑conduc ed he esea ch and has
been ac i e in o ganizing he pape con en , supe ising he anonymiza‑
ion job and he empi ical s udy in sec ion "Da ase desc ip ion". He has also
ca ied ou and w i en he illus a i e case s udy and e ised he pape . AE
has pa icipa ed in he esea ch and is he main w i e o mos sec ions o he
pape . He has also suppo ed some echnical aspec s du ing he empi ical
s udy. JM‑C has ca ied ou he anonymiza ion p ocess, p o ided suppo du ‑
ing he sani iza ion, and has placed he da ase in a public eposi o y. GM has
p o ided echnical suppo wi h he in as uc u e used in he expe imen s
and assis ed in he sani iza ion p ocess.
Funding
This wo k has been pa ly unded by he esea ch G an PID2020‑115199RB‑
I00 p o ided by he Spanish Minis y o Indus y unde he con ac MICIN/
AEI/10.13039 /501100011033.
A ailibili y o da a and ma e ials
Labeled HTTP eques s da ase : Da ase Biblio‑US17. h ps:// doi. o g/ 10. 12795/
11441/ 148254 (2023). [Online; accessed 29‑Jul‑2023].
Decla a ions
Compe ing in e es s
The au ho s ce i y ha hey ha e no a ilia ions wi h o in ol emen in any
o ganiza ion o en i y wi h any inancial in e es o non‑ inancial in e es in he
subjec ma e o ma e ials discussed in his manusc ip .
Au ho de ails
1 Depa men o Signal Theo y, Telema ics and Communica ions, Uni e si y
o G anada, G anada, Spain. 2 Depa men o Telema ics Enginee ing, Uni e si y
o Se ille, Se ille, Spain.
Recei ed: 7 Ma ch 2024 Accep ed: 4 No embe 2024
Re e ences
(2023) Cybe Secu i y S a is ics The Ul ima e Lis O S a s Da a, & T ends Fo
2023 . h ps:// pu pl esec. us/ esou ces/ cybe ‑ secu i y‑ s a i s ics/# Ze oD ay.
Accessed 19‑May‑2023
Abad S, Gholamy H, Aslani M (2023) Classi ica ion o malicious u ls using
machine lea ning. Senso s 23(18):7760
Aga wal N, Hussain SZ (2018) A close look a in usion de ec ion sys em o
web applica ions. Secu Commun Ne w 2018
Page 21 o 21
Díaz‑Ve dejoe al. Cybe secu i y (2025) 8:38
Ageed ZS, Zeeba ee SR, Sadeeq MM e al (2021) A su ey o da a mining
implemen a ion in sma ci y applica ions. Qubahan Acad J 1(2):91–99
Apps SC (2021) Cybe a acks 2021: phishing, ansomwa e & da a b each s a‑
is ics om he las yea . h ps:// spann ing. com/ blog/ cybe a ac ks‑ 2021‑
phish ing‑ anso mwa e‑ da a‑ b each‑ s a i s ics/
Ben Ma in CA, Denis Sinegubko RE, Pelleg ini T (2022) 2022 websi e h ea
esea ch epo . h ps:// sucu i. ne / epo s/. Accessed 28‑Aug‑2024
Be a e G, Giménez E, Ma ínez R e al (2018) Imp o ing web applica ion
i ewalls h ough anomaly de ec ion. In: 2018 17 h IEEE in e na ional con‑
e ence on machine lea ning and applica ions (ICMLA). IEEE, pp 779–784
B ugge ST, Chow J (2007) An assessmen o he da pa ids e alua ion da ase
using sno . UCDAVIS Dep Compu Sci 1:22
Chodak G, Suchacka G, Chawla Y (2020) EClog: HTTP‑le el e‑comme ce da a
based on se e access logs o an online s o e
C e u GF, S a ou A, Locas o ME e al (2008) Cas ing ou demons: sani izing
aining da a o anomaly senso s. In: 2008 IEEE symposium on secu i y
and p i acy (sp 2008). IEEE, pp 81–95
Díaz‑Ve dejo JE, Es epa A, Es epa R e al (2020) A me hodology o conduc ing
e icien sani iza ion o h p aining da ase s. Fu u e Gene Compu Sys
109:67–82
Díaz‑Ve dejo J, Muñoz‑Calle J, Es epa Alonso A e al (2022) On he de ec ion
capabili ies o signa u e‑based in usion de ec ion sys ems in he con ex
o web a acks. Appl Sci 12(2):852
Díaz Ve dejo J, Es epa Alonso RM, Es epa Alonso AJ, Muñoz Calle FJ e al (2023)
Labeled HTTP eques s da ase : da ase Biblio‑US17. h ps:// doi. o g/ 10.
12795/ 11441/ 148254. Accessed 29‑Jul‑2023
Diaz‑Ve dejo J, Es epa R, Alonso AE e al (2024a) Insigh s in o anomaly‑based
in usion de ec ion sys ems usabili y. a case s udy using eal h p
eques s. In: Eu opean in e disciplina y cybe secu i y con e ence, pp
82–89
Diaz‑Ve dejo J, Es epa Alonso R, Es epa Alonso A e al (2024b) Biblio‑us17: a
labeled eal u l da ase o anomaly‑based in usion de ec ion sys ems
de elopmen . In: Eu opean in e disciplina y cybe secu i y con e ence,
pp 217–218
Díaz‑Ve dejo J, Muñoz‑Calle J, Es epa Alonso R e al (2024c) Inspec o log: a
new ool o o line a ack de ec ion o e web log ace iles. In: P oceed‑
ings o he 21s in e na ional con e ence on secu i y and c yp og aphy—
SECRYPT, INSTICC. SciTeP ess, pp 692–697. h ps:// doi. o g/ 10. 5220/ 00127
64000 003767
Es epa R, Díaz‑Ve dejo JE, Es epa A e al (2020) How much aining da a is
enough? A case s udy o h p anomaly‑based in usion de ec ion. IEEE
Access 8:44410–44425
E open (eme ging h ea s open) ule se s (2023) h ps:// commu ni y. eme g
ing h ea s. ne /. Accessed 17‑No ‑2023
Fo bes (2023) Top Websi e S a is ics Fo 2023. h ps:// www. o bes. com/ ad is
o / busin ess/ so w a e/ websi e‑ s a i s ics/. Accessed 19‑May‑2023
Fung BC, Wang K, Chen R e al (2010) P i acy‑p ese ing da a publishing: a
su ey o ecen de elopmen s. ACM Compu Su 42(4):1–53
Gao Y, Ma Y, Li D (2017) Anomaly de ec ion o malicious use s’ beha io s o
web applica ions based on web logs. In: 2017 IEEE 17 h in e na ional
con e ence on communica ion echnology (ICCT). IEEE, pp 1352–1355
Ga cia‑Teodo o P, Diaz‑Ve dejo J, Maciá‑Fe nández G e al (2009) Anomaly‑
based ne wo k in usion de ec ion: echniques, sys ems and challenges.
Compu Secu 28(1–2):18–28
Hajj S, El Sibai R, Bou Abdo J e al (2021) Anomaly‑based in usion de ec ion
sys ems: he equi emen s, me hods, measu emen s, and da ase s. T ans
Eme g Telecommun Technol 32(4):e4240
Hje ppe K, Ruohonen J, Leppänen V (2019) The gene al da a p o ec ion
egula ion: equi emen s, a chi ec u es, and cons ain s. In: 2019 IEEE
27 h in e na ional equi emen s enginee ing con e ence (RE). IEEE, pp
265–275
Husák M, Ap uzzese G, Yang SJ e al (2021) Towa ds an e icien de ec ion o
pi o ing ac i i y. In: 2021 IFIP/IEEE in e na ional symposium on in e‑
g a ed ne wo k managemen (IM). IEEE, pp 980–985
Ib ahim IM, Ameen SY, Yasin HM e al (2021) Web se e pe o mance imp o e‑
men using dynamic load balancing echniques: a e iew. Sys em 19:21
Kh aisa A, Gondal I, Vamplew P e al (2019) Su ey o in usion de ec ion
sys ems: echniques, da ase s and challenges. Cybe secu i y 2(1):1–22
Landaue M, Skopik F, Wu zenbe ge M e al (2020) Ha e i you way: gene a ‑
ing cus omized log da ase s wi h a model‑d i en simula ion es bed. IEEE
T ans Reliab 70(1):402–415
Landaue M, Skopik F, Höld G e al (2022) A use and en i y beha io analy ics
log da a se o anomaly de ec ion in cloud compu ing. In: 2022 IEEE
in e na ional con e ence on big da a (big da a). IEEE, pp 4285–4294
La a‑Rome o A, Te ne o‑Muniz J, Es epa R e al (2023) H p cybe a acks de ec‑
ion h ough au oma ic signa u e gene a ion in mul i‑si e io deploy‑
men s. In: P oceedings o he 2023 Eu opean in e disciplina y cybe secu‑
i y con e ence. ACM, pp 65–70
Mahoney MV, Chan PK (2003) An analysis o he 1999 da pa/lincoln labo a o y
e alua ion da a o ne wo k anomaly de ec ion. In: Recen ad ances in
in usion de ec ion: 6 h in e na ional symposium, RAID 2003, Pi sbu gh,
PA, USA, Sep embe 8–10, 2003. P oceedings 6, Sp inge , pp 220–237
Nemesida ules (2023) h ps:// lin o. nemes ida‑ secu i y. com/. Accessed
17‑No ‑2023
Owasp modsecu i y co e ule se (c s) (2023), h ps:// gi hub. com/ co e ulese /
co e ulese / elea ses/. Accessed 17‑No ‑2023
Pałka D, Zacha a M (2011) Lea ning web applica ion i ewall‑bene i s and
ca ea s. In: A ailabili y, eliabili y and secu i y o business, en e p ise and
heal h in o ma ion sys ems: IFIP WG 8.4/8.9 in e na ional c oss domain
con e ence and wo kshop, ARES 2011, Vienna, Aus ia, Augus 22‑26,
2011. P oceedings 6, Sp inge , pp 295–308
Paxson V (2004) S a egies o sound in e ne measu emen . In: P oceedings
o he 4 h ACM SIGCOMM con e ence on in e ne measu emen , pp
263–271
Rei e MK, Rubin AD (1998) C owds: anonymi y o web ansac ions. ACM
T ans In Sys Secu 1(1):66–92
Salaza ‑He nández R, Díaz‑Ve dejo JE (2010) Anonimización de payloads pa a
el desa ollo de aids basados en p o ocolos. Ac as de las IX Jo nadas de
Ingenie ía Telemá ica (JITEL 2010) pp 260–267
Saxena A, A o a A, Saxena S e al (2022) De ec ion o web a acks using
machine lea ning based u l classi ica ion echniques. In: 2022 2nd in e ‑
na ional con e ence on in elligen echnologies (CONIT). IEEE, pp 1–13
Schulz H, Okano ić D, an Hoo n A e al (2021) Con ex ‑ ailo ed wo kload
model gene a ion o con inuous ep esen a i e load es ing. In: P o‑
ceedings o he ACM/SPEC in e na ional con e ence on pe o mance
enginee ing, pp 21–32
Sha a aldin I, Lashka i AH, Gho bani AA (2018) Towa d gene a ing a new
in usion de ec ion da ase and in usion a ic cha ac e iza ion. ICISSp
1:108–116
Talos ( g oup) communi y ules, (2023). h ps:// www. sno . o g/ alos.
Accessed 17‑No ‑2023
Ue z R, Hemminghaus C, Hacklände L e al (2021) Rep oducible and adap ‑
able log da a gene a ion o sound cybe secu i y expe imen s. In: Annual
compu e secu i y applica ions con e ence, pp 690–705
Viegas EK, San in AO, Oli ei a LS (2017) Towa d a eliable anomaly‑based in u‑
sion de ec ion in eal‑wo ld en i onmen s. Compu Ne w 127:200–216
Publishe ’s No e
Sp inge Na u e emains neu al wi h ega d o ju isdic ional claims in pub‑
lished maps and ins i u ional a ilia ions.
