Mining Web Pages Using Fea u es o Rende ing
HTML Elemen s in he Web B owse
F.J. Fe nández, José L. Ál a ez, Ped o J. Abad, and Pa icia Jiménez*
Abs ac . The Web is he la ges eposi o y o use ul in o ma ion a ailable o
human use s, bu i is usual ha Web Pages do no p o ide an API o ge access o
i s in o ma ion au oma ically. In o de o sol e his p oblem, In o ma ion
Ex ac o s a e de eloped. We p esen a new me hodology o induce In o ma ion
Ex ac o s om he Web. I is based on ende ing HTML elemen s in he Web
b owse . The me hodology uses a KDD p ocess o mining a da ase wi h ea u es
o he elemen s in he Web page. An expe imen a ion o e 10 web si es has been
made and he esul s show he e ec i eness o he me hodology.
Keywo ds: W appe gene a ion, web da a ex ac ion, da a mining.
1 In oduc ion
The Web o e s in e es in o ma ion o human use s h ough Web Pages. How-
e e , in ecen yea s, i s g ow h has gene a ed a la ge olume o in o ma ion o in-
e es , especially in business and, he e o e he esea ch communi y. The aim is
de eloping au oma ic echniques o acili a e he p ocessing o his la ge olume
o in o ma ion o ex end he unc ionali y o adi ional web.
In o ma ion Ex ac ion (IE) om Web Pages ocus on ex ac ing he ele an
in o ma ion om semi-s uc u ed Web Pages, uni y hem and o e s i in a s uc-
u ed o ma o he end-use s. The algo i hms ha pe o m he ask o IE a e e-
e ed o as ex ac o s. Ex ac o s can be classi ied in ou classes [5] acco ding o
he in e en ion o an expe use : hand-c a ed IE Sys ems, supe ised IE Sys-
ems, semi-supe ised IE Sys ems and unsupe ised IE Sys ems.
Unsupe ised IE Sys ems a e based on he hypo hesis ha “Web Pages ha e a com-
mon empla e”. Examples o hese sys ems a e MDR [2], IEPAD [13], RoadRunne
F.J. Fe nández · José L. Ál a ez · Ped o J. Abad · Pa icia Jiménez
Depa amen o In o ma ion Technologies, Uni e si y o Huel a
C a. Huel a-La Rábida, 21819 Palos de la F on e a, Huel a, Spain
e-mail: {ja ie . e nandez,al a ez,ped o.abad,
pa icia.jimenez}@d i.uhu.es
[14], o EXALG [1]. Supe ised IE Sys ems need o anno a ing Web Pages by an expe
use . Examples o hese sys ems a e WIEN [11], So Mealy [4], STALKER [8], WHISK
[12], SRV [7], DEPTA [18], and ViDE [15]. In Semi-supe ised IE Sys ems, he anno a-
ion p ocess can be au oma ed somewhe e: OLERA [3].
The e o e, in supe ised IE Sys ems, he ex ac o s mus que y he Web o col-
lec he Pages, labeling he in e es con en s on he HTML code, and inally, ain-
ing o lea n pa e s ha allow ex ac ing con en o in e es om new Web Pages.
This pape p esen s a new supe ised IE Sys em based on ende ing he HTML
elemen s in he web b owse . Th ee ypes o ea u es ha e been de ined: layou ea-
u es, s yle ea u es and con en ea u es. These ea u es a e calcula ed om HTML
elemen s on he Web Pages. Knowledge Disco e y in Da abases (KDD) p ocess is
applied on his da ase o lea n pa e s o ex ac in e es con en in new web Pages.
The es o he pape is o ganized as ollows: sec ion 2 p esen s he me hodol-
ogy. Sec ion 3 shows he cases o s udy and he esul s. Finally, sec ion 4 con-
cludes his wo k.
2 Me hodology
Almos o he ex ac o s a e based on pa e n ecogni ion o ex ac he in o ma ion
om he s uc u e o he web page. The goal o ex ac o s is o ex ac he ele an in-
o ma ion wi h he minimum e o . Howe e , oday, web pages a e ull o non-
ele an in o ma ion ying o make he web page mo e s iking. Mo eo e , he la ge
numbe o ad e isemen in web pages makes di icul he iden i ica ion o ele an
in o ma ion. In his si ua ion, he ex ac o s mus o deal wi h wo p oblems:
•Fi s , he as amoun o non- ele an in o ma ion o analyze. This p oduces
ha ex ac o s spend a lo o ime wi h he lack o he high compu a ional cos
o hese algo i hms. When he numbe o pages inc eases, he e ec i eness loss
becomes mo e e iden .
Fig. 1 An o e iew o he s eps ha compose he p oposed me hodology
•The second and ha de p oblem is he loss o e ec i eness in he ex ac ion o
he goal in o ma ion. The non- ele an in o ma ion in he pages makes he
web page s uc u e mo e complex and he ex ac o s need o deal wi h hese
complexi y.
The code o he non- ele an in o ma ion in a Web page causes ex ac ion
algo i hms ail when hey y o ex ac he ele an in o ma ion.
The p oposed me hodology is de eloped in he ollowing s eps (Fig. 1):
1. Web Page labeling and ea u e gene a ion. This s ep in ol es he labeling o
he in o ma ion o ex ac and he ea u e gene a ion o his in o ma ion. The
in o ma ion selec ion is pe o med by he use . A e labeling he ea u es o se-
lec ed in o ma ion a e gene a ed and s o ed in a da abase. Fea u es o bo h,
ele an in o ma ion and non- ele an in o ma ion a e gene a ed and s o ed.
2. P ep ocessing o he aining se . Techniques o ea u e selec ion and classes
balancing a e applied o he da abase gene a ed in s ep 1, which is conside ed
as a aining se in a supe ised lea ning p ocess.
3. Models gene a ion. A model o he da a con ained in he aining se is
gene a ed by a supe ised lea ning algo i hm. This model le s classi y new
in o ma ion as belonging o one o he classes.
4. In o ma ion Ex ac ion. The model gene a ed in he p e ious s ep is applied
o new web pages o he same Web si e in o de o ex ac he desi ed da a.
To suppo his me hodology we ha e de eloped he W-SOFIE (Web So wa e
Obse an Fo In o ma ion Ex ac ion) ool. I au oma es he s eps 1 o 3 and
e u ns he induced model om he aining pages.
The nex sec ions desc ibe in de ail each one o he s eps o he me hodology.
2.1 Web Documen Labeling and Fea u e Gene a ion
The Web Pages o in e es a e hose which p esen in o ma ion abou speci ic
p oduc s. These pages display he p oduc de ails as a p oduc in o ma ion shee .
The W-SOFIE ool is able o na iga e h ough hese pages in o de o label he
desi ed elemen s. Labeling is pe o med by he use who also assigns he class o
each selec ed elemen . Once he use has labeled all in e es ing elemen s, he cal-
cula ion o he in e es ing a ea is pe o med. This p ocess in ol es selec ing he
a ea on he page ha con ains he labeled elemen s. Only his a ea will be ea ed,
he es o he page is igno ed because i does no con ain ele an in o ma ion.
The coo dina es o he in e es ing a ea o each page a e s o ed.
The emaining no -selec ed elemen s wi hin he in e es ing a ea a e au oma i-
cally labeled as no -in e es ing. All o hese no -in e es ing elemen s a e assigned
o he same class, he no -in e es ing class.
Once all elemen s on he page a e labeled, W-SOFIE au oma ically gene a es
he ea u es cha ac e ising hem, and hen, a uple is s o ed in he da abase. This
uple ep esen s an elemen and consis s o he calcula ed ea u es and he class as-
signed. Each elemen on he page is s o ed in he da abase, bo h he in e es ing
elemen s and no -in e es ing ones.
Th ee ypes o ea u es a e gene a ed and s o ed o he labeled elemen s:
•Posi ion ea u es. X and Y coo dina es, high and wide o he elemen , he
dis ance om he le - op co ne o in e es ing a ea and he slope o his
hypo he ical line.
•S yle ea u es. Fon colo and s yle, backg ound colo and bo de s.
•Con en ea u es. Tex densi y, digi densi y.
A e his s ep, a da abase con aining a uple o each elemen labeled, and he co-
o dina es o he in e es ing a ea o all ea ed pages is gene a ed. The numbe
o ea ed pages should be su icien o ep esen all he possible layou s o he
p oduc pages.
2.2 P e-p ocessing o he T aining Se
P e ious da abase is eady o KDD p ocess. This da abase is conside ed as a
aining se . Howe e , be o e he unde lying model can be lea ned, a p ep ocess-
ing s ep is necessa y. The da abase has wo main p oblems:
•The da abase has high dimensionali y. The e a e a lo o ea u es ha ep esen
each elemen . In addi ion, much o ha in o ma ion is no ele an . In o de o
sol e his p oblem, se e al ea u e selec ion algo i hms can be used.
•The da abase is imbalanced. The e a e a lo o uples in da abase belonging o
no -in e es ing class, and ew uples belonging o in e es ing class. A balancing
o da abase is equi ed o sol e his p oblem.
A e his s ep, da abase is sui able o an au oma ic lea ning p ocess.
2.3 Models Gene a ion
Because he only elemen s in o in e es ing a ea ha e been conside ed, a lea ning
o he coo dina es o hese a eas is needed. This way, he i s ask o pe o m is
de e mining he coo dina es o his a ea. I is disco e ed om all s o ed a eas o
all labeled pages. An a ea w apping all o hem is conside ed as he gene al
in e es ing a ea.
A e disco e ing he gene al in e es ing a ea, a a ie y o classi ica ion
algo i hms can be used o ex ac a model o he aining da a. The ep esen a ion
o he model depends on he algo i hm used.
I is impo an o highligh ha he lea ning p ocess is pe o med o -line, be-
o e he in o ma ion ex ac ion s ep. Thus, he ime aken o gene a e he model is
spending only once, be o e he in o ma ion ex ac ion p ocess.
2.4 In o ma ion Ex ac ion
The in o ma ion ex ac ion s ep s a s au oma ically ex ac ing all elemen s on a
new page wi hin he in e es ing a ea. The in e es ing a ea conside ed o a new
page is he gene al in e es ing a ea calcula ed in he p e ious s ep. None o all
ex ac ed elemen s ha e a class assigned. In o de o conside hese elemen s as
belonging o a class, hey a e e alua ed using he lea ned model in he p e ious
s ep. The assigned class by classi ie indica es i an elemen is in e es ing o no .
All desi ed elemen s a e s o ed and he no -in e es ing ones a e ejec ed.
3 Expe imen a ion and Resul s
In o de o es he me hodology, i was applied o 10 e-comme ce WEB si es e-
la ed wi h sales o books. These 10 web si es a e he op en by he Alexa
(h p://www.alexa.com) anking.
The 50 Bes selle s pages o each si e we e es ed. The elemen s o ex ac we e:
Ti le, Au ho s, Sa ing, Ra es and A ailabili y.
In o de o educe he dimensionali y o he aining da abase, ou ea u e
selec ion algo i hms we e es ed: In oGainA ibu eE al (w appe algo i hm),
Relie FA ibu eE al (w appe algo i hm), C sSubse E al ( il e algo i hm) and
Consis encySubse E al ( il e algo i hm).
The w appe algo i hms e u n a anking o he ea u es. Ins ead, he il e algo-
i hms e u n a subse o ea u es. In o de o selec a unique subse o ea u es a
hand-c a ed selec ion was pe o med. The ea u es selec ed we e all o hem e-
u ned by il e algo i hms and he mos ele an ea u es (50% o he be e ank)
in he anking o he w appe algo i hms. The mean o he size o he esul ing da-
abase was 18.74% o he o iginal one wi h a s anda d de ia ion o 0.01.
The balancing o he da abase was pe o med by o e sampling; i.e., eplica ing
examples o he mino i y class.
Once p e-p ocessing had inished, ou classi ica ion algo i hms we e es ed:
SMO [10], IBk [6], C4.5 [9] and a S acking me a-model, whe e h ee indi idual
models (SMO, IBk, and JRip [17]) we e combined. The me a-model was ained
om he indi idual models using he C4.5 classi ie .
Table 1 Da ase s be o e p e-p ocessing
Accu acy pe cen age AUC
Web Si es C45 SMO IBK META C45 SMO IBK META
Amazon 99.04 98.92 99.28 99.28 1.00 1.00 1.00 1.00
Ba nesandnoble 98.68 98.52 98.68 97.87 1.00 1.00 1.00 0.94
Bes buy 97.93 99.58 99.17 99.58 0.98 1.00 1.00 1.00
Buy 99.47 99.47 99.47 99.47 0.96 1.00 0.99 1.00
Ebay 98.18 91.64 98.82 99.14 0.97 0.91 1.00 0.99
O e s ock 99.79 99,90 100.00 99.79 0.98 1.00 1.00 0.98
Play 99.71 99.71 99.71 99.41 1.00 1.00 1.00 1.00
Sea s 99.57 99.57 100.00 98.22 1.00 1.00 1.00 1.00
Ta ge 99.30 98.42 98.94 99.30 1.00 0.99 1.00 0.98
Walma 99.66 99.20 99.43 99.31 1.00 1.00 1.00 1.00

Table 2 Da ase s a e ea u e selec ion
Accu acy pe cen age AUC
Web Si es C45 SMO IBK META C45 SMO IBK META
Amazon 99.03 98.92 99.21 99.10 1.00 1.00 1.00 0.99
Ba nesandnoble 98.65 98.57 98.55 98.37 1.00 1.00 1.00 0.99
Bes buy 99.61 99.34 99.30 98.68 0.99 1.00 1.00 0.99
Buy 99.69 99.41 99.47 99.51 0.96 1.00 0.99 1.00
Ebay 98.97 90.47 99.05 99.11 0.98 0.90 1.00 0.99
O e s ock 99.64 99.90 100.00 99.89 0.99 1.00 1.00 1.00
Play 99.70 99.70 99.70 99.53 1.00 1.00 1.00 1.00
Sea s 99.56 99.48 100.00 98.99 1.00 1.00 1.00 1.00
Ta ge 99.74 92.99 98.08 99.39 1.00 0.93 0.99 0.99
Walma 99.73 99.20 99.43 99.47 0.99 1.00 1.00 1.00
Table 3 Da ase s a e ea u e selec ion and balancing
Accu acy pe cen age AUC
Web Si es J48 SMO IBK META J48 SMO IBK META
Amazon 99.96 98.86 99.94 100.00 1.00 1.00 1.00 1.00
Ba nesandnoble 99.86 98.44 99.72 99.89 1.00 1.00 1.00 1.00
Bes buy 99.81 99.75 99.81 99.69 1.00 1.00 1.00 1.00
Buy 99.52 98.78 99.54 99.54 1.00 0.99 1.00 1.00
Ebay 99.94 97.82 99.88 99.93 1.00 1.00 1.00 1.00
O e s ock 100.00 99.98 100.00 99.98 1.00 1.00 1.00 1.00
Play 99.89 99.89 99.89 99.85 1.00 1.00 1.00 1.00
Sea s 99.84 99.73 100.00 99.32 1.00 1.00 1.00 1.00
Ta ge 99.85 99.32 99.71 99.87 1.00 1.00 1.00 1.00
Walma 99.99 99.86 99.91 99.96 1.00 1.00 1.00 1.00
The esul s ob ained by 10- olds c oss- alida ion a e shown in ables 1 o 3.
Values o Accu acy and A ea Unde ROC a e de ailed.
Values on ables show a e y good accu acy and AUC o all algo i hms and
web si es. Mos alues a e nea 100% accu acy and 1 AUC.
Table 4 p esen s he mean o accu acy pe cen ob ained o all classi ica ion al-
go i hms in each web si e.
A s a is ical analysis was ca ied ou conside ing he mean o accu acy pe cen
by means o Wilkoxon es [16]. The es did no de ec signi icance (p=0.05) di -
e ence be ween esul s o da ase be o e p ep ocessing and esul s o da ase wi h
ea u e selec ion ( esul o he es : p=0.492). This way, he conclusion ha ea u e
selec ion does no imp o e he classi ica ion is p obed. Mo eo e , signi icance
(p=0.05) di e ence was ound be ween esul s o Da ase s a e ea u e selec ion
and balancing and he o he s ( esul o he es : p=0.00586 in bo h cases).
The e o e, we can conclude ha ea u e selec ion and balancing p esen s he
bes accu acy.
Table 4 Mean o accu acy pe cen o each web si e
Web Si es Da ase s be o e
p e-p ocessing.
Da ase s a e ea u e
selec ion.
Da ase s a e ea u e
selec ion and balancing.
Amazon 99.13 99.07 99.69
Ba nesandnoble 98.44 98.54 99.48
Bes buy 99.07 99.23 99.77
Buy 99.47 99.52 99.35
Ebay 96.95 96.90 99.39
O e s ock 99.87 99.86 99.99
Play 99.64 99.66 99.88
Sea s 99.34 99.51 99.72
Ta ge 98.99 97.55 99.69
Walma 99.40 99.46 99.93
4 Conclusions
In his pape we ha e p esen ed a new supe ised IE Sys em o ex ac da a in
Web Pages based on ende ing HTML elemen s in he Web b owse . Layou ,
S yle and Con en ea u es a e calcula ed on he elemen s gene a ing a da ase . A
model is induced applying a KDD p ocess o he da ase . Thus, da a om new
Web Pages can be ex ac ed using his model.
Expe imen a ion wi h 10 websi e o e-comme ce has been made. The esul s
show he e ec i eness o his sys em.
The bo leneck o ou me hod is he anno a ion phase, since he use in e en-
ion is equi ed. In u u e wo ks, a semi-au oma ic p ocess will be used.
Acknowledgmen s. This wo k was pa ially sponso ed by Spanish Minis y o Science and
Technology and Jun a de Andalucía unde g an s TIN2007-64119, P07-TIC-02602, and
P08-TIC-4100.
Re e ences
1. A asu, A., Ga cia-Molina, H.: Ex ac ing s uc u ed da a om web pages. In: ACM
SIGMOD In e na ional Con e ence on Managemen o Da a, pp. 337–348 (2003)
2. Liu, B., G ossman, R., Zhai, Y.: Mining web pages o da a eco ds. IEEE In elligen
Sys ems 19, 49–55 (2004)
3. Chang, C.H., Kuo, S.C.: OLERA: Semisupe ised web-da a ex ac ion wi h isual
suppo . IEEE In elligen Sys ems 19(6), 56–64 (2004)
4. Hsu, C.N., Dung, M.T.: Gene a ing ini e-s a e ansduce s o semi-s uc u ed da a ex-
ac ion om he web. In o ma ion Sys ems 23(8), 521–538 (1998)
5. Chang, C.-H., Kayed, M., Gi gis, M.R., Shaalan, K.F.: A Su ey o Web In o ma ion
Ex ac ion Sys ems. IEEE T ansac ions on Knowledge and Da a Enginee ing 18,
1411–1428 (2006)
6. Aha, D.W., Kible , D., Albe , M.K.: Ins ance based lea ning algo i hms. Machine
Lea ning 6(1), 37–66 (1991)
7. F ei ag, D.: In o ma ion ex ac ion om HTML: Applica ion o a gene al lea ning ap-
p oach. In: Fi een h Con e ence on A i icial In elligence, AAAI 1998 (1998)
8. Muslea, I., Min on, S., Knoblock, C.A.: A hie a chical app oach o w ap-pe induc ion.
In: Agen s, pp. 190–197 (1999)
9. Ross Quinlan, J.: C4.5: P og ams o machine lea ning. Mo gan Kau mann Publishe s
Inc., San F ancisco (1993)
10. Pla , J.C.: Fas T aining o Suppo Vec o Machines using Sequen ial Minimal Op i-
miza ion, pp. 185–208 (1999)
11. Kushme ick, N., Weld, D.S., Doo enbos, R.B.: W appe induc ion o in o ma ion ex-
ac ion. In: The 15 h In e na ional Join Con e ence on A i icial In elligence (IJCAI),
pp. 729–737 (1997)
12. Sode land, S.: Lea ning in o ma ion ex ac ion ules o semi-s uc u ed and ee ex .
Jou nal o Machine Lea ning 34(1-3), 233–272 (1999)
13. Csie, T.C., Hui Chang, C.: IEPAD: In o ma ion ex ac ion based on pa e n disco e y.
In: The 10 h In e na ional Con e ence on Wo ld Wide Web, pp. 681–688 (2001)
14. C escenzi, V., Mecca, G., Me ialdo, P.: RoadRunne : Towa ds au oma ic da a ex ac-
ion om la ge web si es. In: The 27 h In e na ional Con e ence on Ve y La ge Da a
Base, pp. 109–118 (2001)
15. Liu, W., Meng, X., Meng, W.: VIDE: A Vision-Based App oach o Deep Web Da a
Ex ac ion. IEEE T ansac ions on Knowledge and Da a Enginee ing 22, 447–460
(2009)
16. Wilcoxon, F.: Indi idual compa isons by anking me hods. Biome ics 1, 80–83 (1945)
17. Cohen, W.W.: Fas E ec i e Rule Induc ion. In: Twel h In e na ional Con e ence on
Machine Lea ning, pp. 115–123 (1995)
18. Zhai, Y.: Web da a ex ac ion based on pa ial ee alignmen . In: P oceedings o he
14 h In e na ional Con e ence on Wo ld Wide Web (WWW 2005), pp. 76–85 (2005)