Abs ac —Non-pa ame ic eg ession models (e.g. ke nel
smoo hing echniques, suppo ec o eg ession model) ha e
been widely used in s a is ics and ecen ly in econome ics and
enginee ing as well. Reg ession unc ions can be e ec i e ools
in he solu ion o enginee ing op imiza ion p oblems. In he
in es iga ion o complex echnical sys ems he class o he
unc ion desc ibing he connec ion be ween he inpu and
ou pu da a is gene ally unknown, mo eo e he “classical”
leas squa e i ing me hod is no lexible enough o p o ide an
e ec i e eg ession unc ion in he case o highe dimensional
op imiza ion p oblems. In his pape eg ession unc ions
ob ained in di e en models a e compa ed g aphically in one
dimensional case. Calcula ions we e made using Maple, R, and
MS Excel so wa e.
Keywo ds—ke nel unc ion, eg ession
I. INTRODUCTION
HE smoo hing echniques and he non-pa ame ic
app oach ha e a long adi ion in empi ical analysis.
The in ensi e in e es in smoo hing o e he p e ious
decades had wo easons: “s a is icians ealized ha pu e
pa ame ic hinking in cu e es ima ions o en does no
mee he need o lexibili y in da a analysis and he
de elopmen o ha dwa e c ea ed he demand o heo y
o now compu able nonpa ame ic es ima es.” [1]
Ke nel smoo hing is an e ec i e eg ession echnique
o enginee ing op imiza ion as well. I has he ad an age
ha he unc ion de ining he non-linea ela ion be ween
he inpu and ou pu da a does no need o be gi en in
explici o m. As an example he shape op imiza ion
p oblem o ubbe bumpe s can be men ioned ha was
in es iga ed by he second au ho in his doc o al hesis
[7].
To p esen he lexibili y o he ke nel smoo hing and
he suppo ec o eg ession we gi e he eg ession
unc ion o ce ain inpu da a (one dimensional case).
II. KERNEL FUNCTIONS
Le us conside a da a se ( equen ly called aining
da e se )
1N
PP11 Rd, ,...,d,
, (1)
whe e
i
a e (possibly one dimensional) inpu ec o s,
i
d
a e associa ed a ge s (
R
is he se o eals). The goal is
o i a unc ion
RR: N
which app oxima es he
ela ion be ween he da a se s. Any eg ession algo i hm
has a loss unc ion
d, L
which desc ibes how he
es ima ed unc ion de ia ed om he ue one. In his no e
we e e o he ollowing wo ypes o loss unc ion: he
quad a ic loss unc ion (leas squa e i ing)
2
iiii d d, L
(2)
and he
-insensi i e (Vapnik’s) loss unc ion
o he wised
d i 0
d, L
ii
ii
ii
, (3)
whe e
is a ixed posi i e pa ame e .
I is well-know ha in he linea model (wi h he
quad a ic loss unc ion)
mindb w w
P
1i
2
ii
(4)
he eg ession unc ion can be exp essed as a linea
combina ion o so-called ke nel unc ions
ii xxk
(5)
in he o m
P
1i ii
op bxkbxwx
, (6)
whe e deno es he inne p oduc [5].
In non-linea models he si ua ion is simila : applying a
sui able ans o ma ion
MN RR:
he p oblem will be
linea in
M
R
and he eg ession unc ion is a linea
combina ion o ke nel unc ions
ii xxk
(7)
[3], [4] in he o m
P
1i ii
op bxkbxwx
. (8)
Since unc ion
and, consequen ly, unc ions
i
k
a e
gene ally unknown in p ac ice, one o he ecommended
APPLICATION OF NON-PARAMETRIC
REGRESSION IN ENGINEERING
OPTIMIZATION
Im e KOCSIS1, Tamas MANKOVITS2
1Uni e si y o Deb ecen Facul y o Enginee ing, [email p o ec ed]deb.hu
2 Uni e si y o Deb ecen Facul y o Enginee ing, amas.man[email p o ec ed]eb.hu
T
ANNALS OF THE ORADEA UNIVERSITY
Fascicle o Managemen and Technological Enginee ing
ISSUE #1, MAY 2013, h p://www.im uo adea. o/auo. m e/
159
ke nel unc ions is chosen in he model o build
. The
mos used ke nel in echnical li e a u e is he adial base
Gaussian ke nel unc ion
2
i
x
iexk
, (9)
whe e
is a pa ame e de e mining he shape o he
ke nel unc ion. The ole o
is p esen ed g aphically in
sec ion III.
III. APPLICATION OF GAUSSIAN KERNEL FUNCTION
As an inpu da a se conside he se o he ollowing
poin s {( i,di)|i=1,..,P}={(1,16),(2,19),(3,9),(4,16),(5,25),
(6,10),(7,5),(8,4),(9,24),(10,20)} (Fig.5.) and sol e he
quad a ic op imiza ion p oblem
mind) (
bebxk)x(
b,
P
1i
2
ii
P
1i
x
i
P
1i ii
2
i
(10)
o
P
P1 R),...,(
and
Rb
, whe e
is a ixed
posi i e numbe .
The ollowing igu es show g aphically he eg ession
unc ions ob ained o ce ain alues o
. (Calcula ions
and plo ing we e ca ied ou using Maple).
05.0
1.0
Fig. 1. The eg ession unc ions (solu ions o he op imiza ion
p oblem (10)) when
=0.05 and
=0.1
15.0
1
Fig. 2. The eg ession unc ions (solu ions o he op imiza ion
p oblem (10)) when
=0.15 and
=1
2
3
Fig. 3. The eg ession unc ions (solu ions o he op imiza ion
p oblem (10)) when
=2 and
=3
I can be seen ha he smalle alues o
esul la e
eg ession unc ions (see e.g. Fig.1.) bu he de ia ion o
hese “ la ” unc ions a he lea ning poin s can be
ela i ely high. La ge alues o
lead o eg ession
unc ions a ying oo as and he shape o he eg ession
unc ions do no seem o be easonable (see e.g. Fig.4.).
Analyzing he pic u es i can be seen ha he alues
2..1
mean an accep able comp omise be ween he
la ness o he unc ion and he accu acy a he lea ning
poin s (Fig.2., Fig.3.).
ANNALS OF THE ORADEA UNIVERSITY
Fascicle o Managemen and Technological Enginee ing
ISSUE #1, MAY 2013, h p://www.im uo adea. o/auo. m e/
160
5
10
Fig. 4. The eg ession unc ions (solu ions o he op imiza ion
p oblem (10)) when
=5 and
=6
IV. SUPPORT VECTOR REGRESSION MODEL
Applica ion o he
-insensi i e loss unc ion (3) leads
o a quad a ic con ex op imiza ion p oblem and he
eg ession unc ion can be exp essed by ke nel unc ions.
In he linea model he eg ession unc ion is sough in
he o m
Rb,Rw,bxwx N
. (11)
The la ness o
in his case means ha one seeks a
small
w
, ha is, he no m
w
is o be minimized. The
ela ed con ex op imiza ion p oblem is he ollowing:
.P,...,1i,
db w
b wd
osubjec
w
2
1
minimize
ii
ii
2
(12)
Inequali ies in (12) mean ha he eg ession unc ion is
equi ed o app oxima e he pai s
ii d,
wi h
p ecision. Some imes he con ex op imiza ion p oblem is
in easible wi h he gi en cons ain s and some “e o s”
mus be allowed. Fo his pu pose so-called slack
a iables was in oduced by Vapnik in [2]. The modi ied
con ex op imiza ion p oblem wi h slack a iables
i
and
'
i
is he ollowing
.P,...,1i,
0',0
'db w b wd
osubjec
'Cw
2
1
minimize
ii
iii
iii
P
1i ii
2
(13)
The posi i e cons an
C
de e mines he ade-o
be ween he la ness o
and he amoun up o which
de ia ions la ge han
a e ole a ed.
Acco ding o he s anda d dualiza ion me hod [6] we
in oduce he Lag ange unc ion
P
1i iii
P
1i iii
P
1i iiii
P
1i ii
2
iiii
)bxw(d''
d)bxw(''
'Cw
2
1
)',,',,b,w(L
(14)
whe e
0',,', iiii
. The dual op imiza ion p oblem
is he ollowing:
].C,0[', and 0' osubjec
''
''
2
1
maximize
ii
P
1i ii
i
P
1i ii
P
1i ii
jijj
P
1j,i ii
(15)
Sol ing he dual p oblem we ge
b x'x i
P
1i ii
, (16)
ha is, he solu ion is a linea combina ion o ke nel
unc ions. This o m o
says ha he explici o m o
w
does no need o be compu ed. Fu he mo e i can be
p o ed ha o he lea ning poin s inside he
- ube
0'
ii
, ha is,
is de e mined by he lea ning
poin s ha ing non anishing coe icien s. These pai s a e
called suppo ec o s.
As i men ioned be o e, in non-linea model he
applica ion o a sui able ans o ma ion unc ion
MN RR:
leads o a linea p oblem. I can be p o ed
ha in his case eg ession unc ion is a linea
combina ion o ke nel unc ions
ii xxk
.
The con ex op imiza ion p oblem is he ollowing
.P,...,1i,
0',0
'db w b wd
osubjec
'Cw
2
1
minimize
ii
iii
iii
P
1i ii
2
(17)
The dual p oblem is
].C,0[', and 0' osubjec
''
''
2
1
maximize
ii
P
1i ii
i
P
1i ii
P
1i ii
jijj
P
1j,i ii
(18)
Sol ing he dual p oblem we ge
b x'x i
P
1i ii
, (19)
ha is, he solu ion is a linea combina ion o ke nel
ANNALS OF THE ORADEA UNIVERSITY
Fascicle o Managemen and Technological Enginee ing
ISSUE #1, MAY 2013, h p://www.im uo adea. o/auo. m e/
161
unc ions.
Using he SVR package o he R so wa e we p esen
he ole o pa ame e s
,
and
C
in he SVR model.
We conside he same inpu da a se as in sec ion II:
{( i,di)|i=1,..,P}={(1,16),(2,19),(3,9),(4,16),(5,25),(6,10),
(7,5),(8,4),(9,24),(10,20)} (Fig.5.) and choose he
Gaussian ke nel unc ion (9). (Plo ing was ca ied ou
using MS Excel.)
Fig. 5. The se o inpu da a
Fig.6. shows he ole o he “penal y” pa ame e
C
.
Applica ion o a highe alue o
C
esul s a eg ession
unc ion wi h small de ia ion e en in ou lying lea ning
poin s, while low alue o
C
gi es a la unc ion wi h
la ge de ia ion a ce ain poin s.
Fig. 6. The ole o he “penal y” pa ame e C
(C=0.1,1,10,
=5,
=0.05)
Fig. 7. The ole o he pa ame e
(
=2,5,8, C=10,
=0.05)
Fig. 8. The ole o he pa ame e
(
=0.05,0.1,0.3,0.5,
=5, C=10)
Fig.7. shows he ole o he pa ame e
(i was
desc ibed also in sec ion II). Fig.8. shows he ole o he
pa ame e
. The lea ning poin s a e possibly
(depending on he alue o he slack a iables) in an
-
ube o nea he
- ube.
Examples p esen ed in his no e show he lexibili y o
he eg ession me hods using ke nel unc ions. In
enginee ing applica ions his lexibili y can ha e an
impo an ole when he model has o be adjus ed o he
special cha ac e is ics o he inpu da a se o o he
equi emen s ela ed o he eg ession unc ion.
ACKNOWLEDGMENT
The desc ibed wo k was ca ied ou as pa o he
TÁMOP-4.2.2/B-10/1-2010-0008 and TÁMOP-4.2.2.A-
11/1/KONV-2012-0041 p ojec s in he amewo k o he
New Hunga ian De elopmen Plan. The ealiza ion o
his p ojec is suppo ed by he Eu opean Union, co-
inanced by he Eu opean Social Fund.
REFERENCES
[1] W. Ha dle, “Applied non-pa ame ic eg ession”, Humbol -
Uni e si ä , Be lin, 1994
[2] C. Co es and V. Vapnik, “Suppo ec o ne wo ks”, Machine
Lea ning, Vol. 20, pp. 273-297, 1995.
[3] B. Schölkop , and A. J. Smola, “Lea ning wi h Ke nels”, MIT
P ess, 2000
[4] A. Fa ag and R. M. Mohamed, “Reg ession Using Suppo Vec o
Machines: Basic Founda ions”, Uni e si y o Louis ille, 2004
[5] S. Haykin, “Neu al Ne wo ks and Lea ning Machines”, P en ice
Hall, 2009
[6] S. Boyd and L. Vandenbe ghe, “Con ex Op imiza ion”,
Camb idge Uni e si y P ess, 2009
[7] T. Manko i s, “Shape op imiza ion o ubbe pa s”, PhD Thesis,
Uni e si y o Miskolc, 2013
ANNALS OF THE ORADEA UNIVERSITY
Fascicle o Managemen and Technological Enginee ing
ISSUE #1, MAY 2013, h p://www.im uo adea. o/auo. m e/
162
