Searching a database based web site

Sea ching a Da abase Based Web Si e Filipe Sil a1 and Gab iel Da id1,2 1 Faculdade de Engenha ia da Uni e sidade do Po o Rua D . Robe o F ias 4200-465 Po o Po ugal, { sil a,g d}@ e.up.p h p://www. e.up.p / 2 Inesc-Po o Rua D . Robe o F ias 4200-465 Po o Po ugal Abs ac . Cu en ly, in o ma ion sys ems a e usually suppo ed by da abases (DB) and accessed h ough a Web in e ace. Pages in such Web si es a e no d awn om HTML iles bu a e gene a ed on he ly upon eques . Indexing and sea ching such dynamic pages aises se e al ex a di icul ies no sol ed by mos sea ch engines, which we e designed o s a ic con en s. In his pape we de- sc ibe he de elopmen o a sea ch engine ha o e comes mos o he p oblems o a speci ic Web si e, how he limi a ions pu o indexing dynamic Web pages we e ci cum en ed, and an e alua ion o he esul s ob ained. The solu ion in- ol es using a locally de eloped c awle , he O acle Tex ull ex indexe , and me a-in o ma ion au oma ically d awn om he DB o manually added o im- p o e he ele ance ac o calcula ion. I has he ad an age o uni o mly co e - ing he dynamic pages and he s a ic Web pages o he si e. Keywo ds. in o ma ion e ie al, Web sea ch engine, indexing da abase-based Web si es 1 In oduc ion The e olu ion b ough by he Web o he in o ma ion wo ld is no as much due o he huge olume o in o ma ion a e e yone’s disposal as o he d ama ic inc ease on he in o ma ion access e iciency ha made possible p e iously un easible asks [1]. Among he key ools in his con ex a e he Web sea ch engines. They e ol ed om an idea as simple as he model o he Web: he Web is a ne wo k o Web se e s, each one esponsible o a collec ion o c oss-linked HTML pages (see Fig. 1). The sea ch engine includes a c awle ha , s a ing a a gi en URL, e ie es he co e- sponding page and ecu si ely ollows i s links o o he pages. Then i indexes he ull ex o he page, along wi h i s URL. A e wa ds, i becomes able o answe eques s on in o ma ion ela ed o speci ied wo ds by e u ning he URLs o he pages con ain- ing hem. Thus, he sea ch engine can be seen as made o h ee componen s: he c awle , he indexe , and he que y p ocesso . Many de elopmen s o his basic idea ha e been p oduced in o de o imp o e he ele ance ac o o each page and he p ecision and ecall me ics o he sea ch [2], o ins ance, coun ing he numbe o occu ences o each wo d in he page, assigning mo e weigh o occu ences in head- ings o in META ags, combining wi h o he me hods o classi ica ion, e c. Fig. 1. Answe ing a s a ic page eques Howe e , he Web con ains mo e han collec ions o HTML pages. In o ma ion sys ems (IS) con ain la ge amoun s o in o ma ion which is ypically mo e s uc u ed han he ex s in HTML iles and is o ganized in da abase sys ems wi h hei own link- ing mechanisms. Cu en IS a e accessible ia in e aces ha adop ed he Web pa a- digm and echnology. The end-use jus equi es a Web b owse o na iga e h ough he pages o he co esponding si es. The main di e ence is ha hese pages no longe co espond o a HTML ile s o ed in a ile sys em bu a e ins ead gene a ed upon a use eques by speci ic p og ams ha ga he he in o ma ion needed om he da abase and deli e he HTML page di ec ly o he use (see Fig. 2). These pages a e called dynamic pages1 in da a in ensi e si es [3] as opposed o he s a ic pages ozen in HTML iles. The e a e a numbe o ad an ages o his app oach. The quali y o he in o ma ion p esen ed di ec ly bene i s om he abili y o DB sys ems o o ganize, main ain consis- ency, upda e and con ol he access o in o ma ion. Gene a ing he page a eques - ime enables a mo e lexible p esen a ion o in o ma ion, h ough he speci ica ion o se e al sea ch c i e ia, and e en o pe sonalize i , i he use is known o he sys em. 1 This is dis inguished om pages ha inco po a e Dynamic HTML o Ja aSc ip o ge ani- ma ion e ec s bu a e ne e heless de ined be o ehand in a s a ic ile. Fig. 2. Answe ing a dynamic page eques The use is p obably no awa e o hese echnical issues and sees he si e as a se o linked pages i espec i e o hei sou ce, eal o “ i ual” iles. So, he use expec s a sea ch acili y ha answe s a se o ele an pages o a que y in ol ing one o mo e e ms. This goal o uni o mly dealing wi h s a ic and dynamic si es is especially ele- an in mixed si es combining s uc u ed da a wi h non-s uc u ed ex . This is he case o he in o ma ion sys em o he Facul y o Enginee ing o he Uni- e si y o Po o (SiFEUP, [4,5]), an in eg a ed sys em ha combines academic eco ds o a ious kinds wi h class summa ies, bibliog aphic da a wi h eaching ma e ials, e c. SiFEUP is he case s udy conside ed in his pape . The ollowing sec ions p esen in mo e de ail he p oblems aised o he sea ch en- gines by dynamic Web si es, he solu ion ha has been de eloped, and an assess- men o he esul s achie ed. 2 Why Sea ch Engines Fail in Dynamic Web Si es Be o e he discussion o he abo e men ioned di icul ies, he e is a compa ison wi h he usual way o que ying an IS wi h a Web in e ace. Depending on he sophis- ica ion o he si e, he in e ace o he DB may be as ob ious as a se ies o o ms whe e he use ypes he sea ch c i e ia ecei ing as an answe a page wi h a able con aining he eco ds which ha e been e ie ed by he DB que y p ocesso . In some sys ems i is e en possible o w i e down an SQL que y. The main p oblems wi h he o ms-based app oach a e: − he o ms o be used mus be an icipa ed by de elope s, ying o ma ch he ypical use eques s bu lea ing ou mo e speci ic needs; − in a DB, he in o ma ion is s o ed in di e en ables and columns and e ie al mus speci y which one is o be used, bu use s don’ like o choose ields be o e he sea ch [6]; − he sea ch equi es exac -ma ch and he SQL o simula e a gene ic sea ch (including s emming and he s ipping o capi aliza ion and accen s) becomes a he clu e ed and complex [7]; − all he eco ds sa is ying he sea ch c i e ia a e e ie ed wi h equal impo ance and o de ed by column con en s, bu he ele ance o he i ems in he answe a e di e - en and acknowledging his is c ucial when la ge numbe s o eco ds a e e ie ed [6]. No e e y hing is bad wi h he o ms-based que y wi h espec o he ull- ex index- ing: − DB indexing is usually done in eal- ime, while in he o he case i is de e ed and hus equi e pe iodic upda e; − he DB needs less compu ing esou ces (CPU and s o age) han he sea ch engine; − he ou pu so ing me hod can be speci ied, while in he sea ch engine is usually ixed o be he dec easing ele ance o de . In mo e wo ked in e aces, he abula na u e o he sys em is much mo e dis an om he use eyes, and he page includes signi ican pieces o ex and da a collec ed om se e al DB ables, c ea ing some hing close o a documen han o a DB ex ac . A gene ic sea ch acili y is expec ed o e ie e hese documen s and no lis s o e- co ds om he DB. The combina ion o he Web and DB echnologies p oduce highly mu able pages which may be di e en depending on he use and on he momen o he eques . So, he e y no ion o documen mus be e ised. In his pape , i o simi- la eques s he sys em answe s pages wi h di e en con en s because he use is di e en , hey a e conside ed di e en documen s. I he di e ence is only due o changes in he DB be ween he eques s, hese a e conside ed di e en e sions o he same documen . The e o e a gene ic sea ch mechanism is jus i ied. Howe e , ypical sea ch engines, which we e designed o s a ic con en s, do no index dynamic Web pages o jus do ha o hose ha a e speci ically linked om a s a ic page and do no ollow on he links he dynamic page may con ain. Indexing and sea ching dynamic pages aises se e al ex a di icul ies no sol ed by mos sea ch engines [8,9,1]: − he numbe o URLs o dynamic pages in a si e may be in ini e due, o example, o he use o a session numbe as an en y pa ame e in he URL, which changes on each eques , despi e he ac ual con en s emaining he same; − he e is he possibili y o alling in o an in ini e loop o page gene a ion no easy o de ec ; − many dynamic pages a e no di ec ly accessible om a link in ano he page bu only as a esul o he submission o a o m illed in wi h app op ia e alues, bu allowing he c awle o submi a o m is ques ionable because i is no easy o know which alues o choose ( he numbe o combina ions may be in ini e) and some imes o submi a o m causes a change in he DB, some hing a c awle is no supposed o do; − he inhe en dynamics o hese pages leads o indexes buil by he sea ch engine ha become ou da ed a a as pace; − in si es wi h access con ol, he c awle s a e assigned a gene al public s a us, e- ducing he in e es o he engine o he quali ied use ; − i is ha d o decide on index e esh policies due o he absence o a iabili y o las change da e. In conclusion, a signi ican numbe o he pages ha a use may be in e es ed in a e no indexed by mos sea ch engines. They cons i u e pa o wha is called he In isi- ble o Hidden Web [10,11,12,13,14,15], along wi h he pages ha ha e no access pa h o hem. 3 Con ibu ion o Reducing he In isible Web Among he echniques o o e come some o he ba ie s lis ed in he las sec ion one is he analysis o he Web o ms ha he c awle goes in o. I looks o inpu ields wi h associa ed lis s o alues which may be epea edly used in au oma ic submis- sions o ha o m, in o de o collec he maximum numbe o esul pages [16,17,18,19,20]. Going much u he equi es he use o me a-in o ma ion abou he DB. This is he app oach ollowed in his pape because he mo i a ing case s udy p oblem is a local o ganiza ional Web IS. The e o e, he e is knowledge abou he DB schema and access o he DB con en s, which can be used by he sea ch engine. This si ua ion is common o o he in ane IS. The solu ion desc ibed in his sec ion p oposes an a chi ec u e whe e he inside in o ma ion needed is clea ly iden i ied and localized. The p oposed sea ch engine (see Fig. 3) includes he usual componen s ( he c awle , he indexe , and he que y p ocesso ) plus a DB in o ma ion p ocesso , which concen a es he DB speci ic knowledge equi ed o imp o e he ecall. The c awle is a complex componen accomplishing se e al asks. The Page Finde is in cha ge o ollowing he links and does ha wi h no o he conce n besides no going ou side he a ge URL domain. As opposi e o many c awle s, i accep s URL pa ame e s and insis s on ollowing e e y link o any dep h, leading bo h o dynamic and s a ic pages. The a ge domain and excep ional URLs ha mus no be ollowed a e s o ed in he Con igu a ion Da a. The s a ing poin s a e s o ed in he S a ing URL lis . The links ound in he p ocessing o a documen a e queued in he Docu- men s URL lis , o a la e isi . Fig. 3. Sys em a chi ec u e The ideas behind dealing wi h o ms a e: alues in hidden ields o a o m, no mally used o o wa d p e iously collec ed da a, a e kep as hey a e and used in he au o- ma ic submission o he o m; he o he ields a e illed in wi h he de aul alues, i any, o aken o be null. Combina ions o possible alues p esen in combo-boxes and adio bu ons a e no cu en ly used as, in SiFEUP, hey would mean o ob ain a sub- se o he de aul answe . The Fo m P ocesso will apply his echnique o he o ms classi ied as able o be ollowed by a me a- ag in he page i sel . This is an example o es ablishing some ules o he de elopmen o new modules in he IS. The Upda e Con olle uses HTTP in o ma ion (I -changed-since ag) o ecognize a change in a s a ic documen , and he in o ma ion gi en by he DB In o ma ion P oc- esso o he dynamic pages. The S a ic Page Desc ip o ge s me ada a abou he s a ic pages. The me ada a ela i e o he dynamic pages comes om he DB In o ma ion P ocesso as well. This in o ma ion is s o ed in he Documen Me ada a able. Among o he da a, he size o he documen is a ailable. The las module in he C awle componen is he Tex P ocesso . I cleans up he ex , o ins ance s ipping HTML ags, and con e s om di e en ile o ma s like MS O ice, PDF, e c. The o he modules ha e been speci ically de eloped o his p ojec . Howe e , o he Tex P ocesso and he Indexe , an O acle Tool called O acle Tex has been used, and p ope ly con igu ed. The Indexe builds wo in e ed ile indexes whose en ies a e he wo ds appea ing espec i ely in he documen and in he me ada a (Documen Me ada a), excep o he s op-wo ds lis ed in he co esponding able. The Que y P ocesso is he hi d componen . I ecei es a use que y, may change i using he synonyms dic iona y and he s op wo ds, and consul s he indexes o build he esul , acco ding o he ele ance in o ma ion o he pages. To calcula e he ele ance ac o , he s a ing poin is he alue gi en by O acle Tex . This numbe lies be ween 0 and 100 and is gi en by 3 (1 + log( N/n )) whe e is equency o he e m in he documen ; N is he o al numbe o documen s and n is he numbe o documen s con aining he e m. This way, a e m showing up in many documen s is less ele an han a e m con ained in ewe documen s. Howe e , O acle Tex is no able o index a he same ime he HTML ex ead and he ex a me a ags p oduced by he me a-desc ip o o he dynamic pages. So, a modi- ica ion o he ele ance ac o has been done o accoun o his and o he impo - ance o he page in i s en i onmen . The i s componen is based on he con en s o he wo indexes and is weigh ed be ween 0 and 30. The second componen gi es mo e ele ance o he pages based on ables which a e cen al and possess mo e ela ion- ships going in o hem. This ac o is weigh ed ypically be ween 0 and 15 and is e- sponsible o gi ing mo e ele ance o he home page o a p o esso han o his lis o publica ions, whe e his name is likely o appea a lo mo e equen ly. To comple e he desc ip ion o he sys em de eloped, he DB In o ma ion P oces- so ge s me ada a abou he dynamic pages, including c i e ia o suppo he e esh- ing policy and imp o ing he pages ele ance ac o . I also deals di ec ly wi h DB columns ha a e known, in he Con igu a ion Da a, o con ain URLs, like he links o he eache o he s uden non-o icial pages. As he RDBMS used by he case s udy IS is O acle, he language chosen o he de elopmen has been PL/SQL. The es s pe o med wi h he sys em indexed abou 140 000 pages, bo h dynamic and s a ic, including non HTML documen s. The sample es que ies used we e aken om a wo-days log o he ac ual sea ches pe o med on he main sys em o ms by he use s. Addi ionally some expe imen a ion has been made by expe s, o mo e complex and c i ical que ies. These included ope a- o s like AND, OR and dis ance in he ex be ween ce ain speci ic e ms. The answe ime o que ies a ied om negligible o 10 seconds, wi h a ypical alue a ound 3 seconds. Documen s ound s answe ing ime 0 5 10 15 20 0500 1000 1500 2000 2500 3000 Documen s ound Answe ing ime (s) Fig. 4. Sea ch esul s The analysis o he g aphic in Fig. 4 shows ha some que ies p oduce la ge quan i- ies o documen s. They co espond o speci ying a common single e m que y. This is no a bad esul om he iewpoin o inding he app op ia e documen s, because he mo e ele an a e shown i s . Howe e , i implies a longe compu a ion ime while he use is wai ing, spen mainly on so ing he esul . Fo una ely, mos que ies lie on he le side o he g aphic meaning sho e and, mos o he ime, as e answe s o 3-4 e m que y. 4 Conclusion Cu en IS a e subjec o an appa en con adic ion. On one hand hey a e buil on op o da abases s o ing highly s uc u ed in o ma ion. On he o he hand, he p esen- a ion o his in o ma ion is done ia Web in e aces made o HTML pages, usually no s uc u ed. The b idge be ween he DB and he use is made by dynamic Web si es, in he o ganiza ional in ane which is, some imes, accessible by he whole In e - ne . Using complemen a y in o ma ion om he RDBMS ela ionships in he pages gene a ed om ha da a, a be e ecall igu e is ob ained. The main conclusion is ha adding o he IS he gene ic ex ual sea ch composed wi h me ada a om he DB imp o es he access o in o ma ion, wi h espec o he adi ional me hod based on menus, hype links and Web o ms, especially o he occasional use s. I has been no iced ha , e en o egula use s, i inc eases he isi- bili y o ce ain con en s in he pe iphe y o he sys em. The c i e ia ollowed in modi ying he ele ance ac o , hough always subjec i e, p o ed o b ing o he i s ows he kinds o pages he use expec s. Fo example, asking o a pe son’s name e ie es, mos o he imes, he co esponding o icial pe sonal page. This happens because hose pages a e cen al in he sys em. Some- imes, he i s ow is he bibliog aphic page o ha pe son, as i con ains mul iple occu ences o he name as he au ho o he di e en publica ions. Finding he desi - able weigh ac o s equi es a ca e ul adeo . The esul s o he in e ac ion wi h an In o ma ion Re ie al sys em a e o a di e en na u e om hose ob ained om a da abase. When que ying a DB, he answe gi en o he use is a eco d se co esponding, in a de e minis ic and objec i e way, o he que y, i espec i e o he DBMS used. In an in e ac ion wi h an In o ma ion Re ie al sys em, he answe s a e less de e mined by he que y, hough some le el o ela ion- ship always exis s. De e minism in hese sea ches is de i ed mo e om he na u e o he documen p ep ocessing echniques like index cons uc ion and clus e ing, and que y in e p e a ion, han om he s uc u e o he in o ma ion. Di e en sys ems ypically p oduce esul s no ully coinciden . Re e ences 1. LAWRENCE, S e e ; GILES, C. Lee - Accessibili y o In o ma ion on he Web. Na u e. 400, no. July 1999 (1999) 107-109. 2. JONES, Ka en Spa k ; WILLETT, Pe e - O e al In oduc ion. In Ka en Spa k Jones e Pe e Wille , eds.- Readings in In o ma ion Re ie al. Mo gan Kau mann, 1997, 1-7. 3. FRATERNALI, P. - Tools and app oaches o de eloping da a-in ensi e {Web} applica- ions: a su ey. ACM Compu ing Su eys. 31, no. 3 (1999) 227-263. h p://www.ucsd.edu/cse132B/WSMT.pd . 20-12-2002. 4. DAVID, Gab iel ; RIBEIRO, Lígia Ma ia - Ge ing Managemen Suppo om an Uni e - si y In o ma ion Sys em. P oceedings o he Eu opean Coope a ion in Highe Educa ion In- o ma ion Sys ems, EUNIS99, Espoo, Finland, 1999.