Human s. Compu e Slo Ca Racing using an E en and F ame-Based
DAVIS Vision Senso
T. Delb uck1, M. P ei e 1, R. Jus on2, G. O cha d3, E. Müggle 4, A. Lina es-Ba anco5, M.W. Tilden6
1: Ins i u e o Neu oin o ma ics, Uni . o Zu ich and ETH Zu ich, Swi ze land,
2: Bio obo ics Team, Ins i u e o Mo emen Sciences, CNRS / Aix-Ma seille Uni .
3: Singapo e Ins i u e o Neu o echnology (SINAPSE)
4: Robo ics and Pe cep ion G oup, Uni . o Zu ich
5: Robo ics and Technology o Compu e s Lab, Uni . o Se ille
6: Design consul an , Wowwee L d, Hong Kong
Abs ac –This pape desc ibes an open-sou ce
implemen a ion o an e en -based dynamic and
ac i e pixel ision senso (DAVIS) o acing
human s. compu e on a slo ca ack. The
DAVIS is moun ed in "eye-o -god" iew. The
DAVIS image ames a e only used o se up and
a e subsequen ly u ned o because hey a e no
needed. The dynamic ision senso (DVS) e en s
a e hen used o ack bo h he human and
compu e con olled ca s. The p ecise con ol o
h o le and b aking a o ded by he low la ency o
he senso ou pu enables consis en ou -
pe o mance o human d i e s a a lap op CPU
load o <3% and upda e a e o 666Hz. The spa se
ou pu o he DVS e en s eam esul s in a da a
a e ha is abou 1000 imes smalle han om a
ame-based came a wi h he same esolu ion and
upda e a e. The scaled a e age lap speed o he
1/64 scale ca s is abou 450km/h which is wice as
as as he as es Fo mula 1 lap speed. A eedback-
con olle mode allows compe i i e acing by
slowing he compu e con olled ca when i is
ahead o he human. In es s o human s.
compu e acing he compu e s ill won mo e han
80% o he aces.
I. INTRODUCTION
The DAVIS is a neu omo phic came a ha
ou pu s s a ic image ames concu en ly wi h
dynamic ision senso (DVS) empo al con as
e en s [1][2]. DVS add ess-e en s (AEs)
asynch onously signal changes o log in ensi y. The
AE imes amp (in mic oseconds) codes he ime o he
e en s. Pixels wi h DVS ou pu a e neu omo phic
abs ac ions o e inal ganglion cells in biological
e inas. Thei sub-ms la ency, spa se ou pu , and kHz
pixel bandwid h has led o applica ions equi ing high
speed objec acking wi h sho -la ency eedback, e.g.
[3][4]. In his wo k, we use he DVS ou pu s o ack
slo ca s and con ol one o he ca s o ace
compe i i ely agains human d i e s. In his
applica ion, he DAVIS s a ic ames we e use ul o
se ing up he senso and adjus ing ocusing, bu
subsequen ly he ames we e no needed and we e
u ned o .
II. HARDWARE AND SOFTWARE SETUP
Fig. 1 shows a ace ack oge he wi h sample DVS
da a p oduced by a single ca d i ing a ound he ack.
The 240x180 pixel DAVIS was moun ed in “eye o
god” iew o e he able using a wide angle 2.6mm
lens wi h a ho izon al ield o iew o 81 o co e he
slo ca ack. As he ca s go a ound he ack, hey
c ea e DVS e en s, which a e shown supe imposed as
3D space- ime e en s o e he pho o o he ack.
These e en s a e used as desc ibed la e o ack he
ca s and o con ol he compu e ca h o le and
b aking. The e en s cap u ed om he DAVIS a e
ansmi ed o he hos PC o e a USB in e ace.
Indi idual e en s a e ime-s amped wi h 1us
esolu ion.
Slo ca s con ain a DC mo o , a pin o guide he
ca along he slo in he ack, wo b ass b ushes ha
pick up powe om he me al ails o he ack, and a
magne ha helps hold he ca on o he s eel ack
ails. Powe o he ca is no mally egula ed by a
simple h o le con olle consis ing o a wi e-wound
esis o . Race s a emp o go as as as possible
a ound he ack wi hou lying o i . We used a HO-
scale (1/64) sys em om AFX Racing
Fig. 1. A ack layou wi h he s eam o DVS e en s (do s)
caused by a mo ing ca shown as do s in 3D space- ime.
The a e age e en a e caused by a mo ing ca is abou 5k
e en s/sec.
(www.a x acing.com) wi h ca chassis ype
SRT. The ca s a e 7cm in leng h, and he
ack (Fig. 2) had 15 u ns in a leng h o
805cm, o 900 pixels on he senso image.
The as es lap imes a e abou 4.1s. The
slo ca speed o 200cm/s scaled o
Fo mula 1 size is 450km/h; o e e ence,
he as es a e age lap speed achie ed in a
Fo mula 1 ace was 248km/h on a ack
wi h 11 u ns (Monza 2003). Ca s can
accele a e o ull speed in abou 300ms and
decele a e by ic ion in abou he same
ime. The elec onic mo o b aking
(desc ibed la e ) slows he ca e en as e ,
allowing mo e agg essi e d i ing.
Implemen a ion
The so wa e implemen a ion o he slo
ca ace is open-sou ced in he jAER
p ojec [5] in he package
ch.unizh.ini.jae .p ojec s. i ualslo ca [6]. The main
slo ca ace class is Slo Ca Race . The h o le
con olle desc ibed in his pape is he class
HumanVsCompu e Th o leCon olle .
1) Ca T acking, T ack Model, and T ack
Masking
The ope a o sees a iew o he ack and o he
in o ma ion supe imposed on he DAVIS senso
ou pu as shown in Fig. 2. Di e en pa s o his
display a e labeled and a e e e ed o below.
T acking (compu ed by he class Ca T acke ) uses
a model o he ca consis ing o a ec angle ha is
cons ained o mo e along he ack model, which is a
lis o ack e ices in senso pixel coo dina es. A
Ca Clus e is a so wa e objec based on [4] ha has a
2D pixel posi ion and a eloci y along he ack in
ack e ices pe second. The ack model is ob ained
in a semi-au oma ed way in T ackDe ineFil e by
d i ing each ca a ound alone, collec ing a 2D e en
his og am, and hen ex ac ing a lis o e ices spaced
by minimum dis ances, s a ing om he peak o he
his og am. This lis is hen upda ed manually using a
GUI o d ag, add, and dele e e ices.
Fig. 4 illus a es he ca acking upda e. Each
DVS e en inpu o Ca T acke is i s used o upda e
he cu en ca posi ion acco ding o he ca eloci y
along he ack, using he las upda e ime and he
cu en e en ime. This upda e implemen s he model
ine ia. Then he e en is checked i i is nea he
loca ion o he acked ca (shaded ec angle in Fig. 4
and bounda ies o boxes su ounding ca s in Fig. 2).
I so, he ca model is upda ed by ei he ad ancing o
e a ding he ca posi ion along he ack depending
on whe he he DVS e en leads o lags he cu en
ca posi ion. The amoun o ad ancemen o e a ding
is se by a ‘mixing ac o ’ ( ypically abou 0.02) ha
mixes he DVS e en posi ion wi h he cu en ca
posi ion wi h he mixing ac o p opo ion. The upda e
is done by p ojec ing he ec o e om cu en ca
posi ion o he e en on o he ack ec o
connec ing he nea es ack e ex o he nex one
along he ack. A 2D lookup able (Fig. 3) maps pixel
coo dina es o he nea es ack e ex, o speed up he
sea ch o he nea es ack e ex. The ca ’s ack
eloci y is upda ed when he nea es e ex changes.
The ca acke li e ime is managed by a ‘mass’ ha
decays away exponen ially wi h ime be ween DVS
e en s and is inc emen ed wi h each e en . I he mass
Fig. 2. Display o he slo ca ack wi h s a e in o ma ion.
Fig. 3. Mapping om each pixel loca ion o he nea es ack
e ex speeds up lookup o he nea es ack e ex
(numbe a each pixel loca ion). Fig. 4. Ca T acke ca posi ion upda e.
alls below a ce ain alue, he ca is conside ed o be
los and acking mus be eini ialized.
2) Slo Ca Th o le and B aking Ha dwa e
We designed a slo ca con olle PCB (Fig. 5) o
con ol he powe o up o 4 slo ca acks om a
compu e o e a ull-speed USB2.0 in e ace. This
con olle con ols powe o he ca s by 1.5kHz PWM
modula ion o he 17V ack powe , and also can sho
he ca mo o ac oss a 20 esis o o elec onic
b aking. Only one ack con olle is cu en ly used
and he o he ack is con olled by he human using
he s anda d h o le. The con olle is upda ed wi h
polling in e al o 1ms. The ull PCB design o his
con olle and i s i mwa e a e a ailable in he jAER
p ojec [7]. A p o o ype design did no use
op ocouple s and he esul s we e equen ese s o
he USB mic ocon olle due o la ge ol age
ansien s caused by spa king o he ca con ac o he
ack which p opaga ed back h ough he elec onics.
The inal design o Fig. 5 uses op ocouple s o co ec
his p oblem.
3) Th o le Con ol
Con olling he compu e ca consis s o se ing he
h o le alue o applying he b ake a each e ex o
he ack model based on a ec o o h o le/b ake
se ings called a h o le p o ile. An example h o le
p o ile is shown in Fig. 6. We in es iga ed a numbe
o me hods o op imize he h o le p o ile. E en ually
we ound ha he as es me hod is o 1) de e mine an
ini ial h o le p o ile by se ings de i ed au oma ically
om he ack cu a u e so ha s aigh sec ions ha e
highe ini ial h o le; 2) examining he ca isually
and hen g adually inc easing he h o le o applying
b ake by eye, using a GUI in e ace o “pain ” h o le
and b ake se ings. An e olu iona y me hod was also
de eloped o lea n he op imum h o le p o ile by
inse ing h o le inc eases and seeing i hey esul in
success ul laps. A e a c ash, he inse ion is emo ed
o b aking poin s a e inse ed. The equi ed lea ning
ime is cu en ly s ill conside ably longe han by
manual adjus men o he p o ile and mo e wo k
needs o be done o unde s and he op imum s a egy.
To make acing mo e compe i i e, a mode can be
enabled ha slows he compu e down om i s
op imum h o le alue o a minimum alue,
depending on how a ahead is he compu e ca .
Typically a alue o one hi d o he o al ack o
comple e slow-down is e ec i e o esul ing in
exci ing side-by-side acing.
III. RESULTS
A se ies o YouTube ideos documen he
e olu ion o he slo ca ace since 2010 [8]–[10].
The inal ideo shows he se up and a ace be ween
compu e and human, including con ol ha slows he
compu e ca when i is ahead o he human.
A sample o wo eco ded laps by he compu e
con olled ca is shown in Fig. 7. This da a is aken
om a di e en ack. O e wo laps, he ca posi ion
inc eases and hen w aps back o e ex 0. A he end
o he s aigh way, mo o b aking apidly dec eases
he ca speed, as indica ed by he “b aking” a ow.
Du ing he las lap, mo o b aking is disabled (“No
b aking”), esul ing in a c ash a e he s aigh way.
A se ies o en 3-lap aces be ween human and
compu e had he ollowing esul s: The i s human
had 1 win, 2 losses, and 7 DNF (did no inish, i.e.
c ashed). The second human had 3 wins, 4 losses, and
3 DNFs, howe e a e he eedback con ol o slow
he compu e ca was u ned o hal way h ough he
se ies o en aces, he second human was no longe
able o win.
P ocessing cos and h oughpu : P ocessing
Slo Ca Race on a Leno o W510 Co e i7 lap op
esul s in a CPU load o 1% o 3% o he Ja a i ual
machine, when g aphical ende ing is disabled.
The upda e in e al on he hos compu e was
de e mined by ins umen ing he USB da a packe s
ecei ed by he high-p io i y USB p ocessing h ead
Fig. 5. The slo ca con olle PCB con ols up o 4 lanes [7].
A: in e ace ci cui o a single lane o h o le and b ake
con ol. The op ocouple pulldown ou pu s (Op Th Ou &
Ou B Ou ) eed he powe MOSFET ga es h ough RC low
pass il e s wi h=0.5ms. A use F1 p o ec s agains
sho s. R16 is a powe esis o o mo o b aking. B:
Fab ica ed PCB wi h USB cable (bo om) and ack/powe
connec ions ( op).
Fig. 6. Th o le and b ake p o ile ha achie es 4.1s lap imes on
ack in Fig. 2.
using he Ja a me hod
Sys em.nanoTime(). All he slo ca ace
p ocessing is done in his h ead, a he
han he display ende ing h ead, which
uns a mos 60Hz. The DAVIS came a
includes a ea u e called ‘ea ly packe
ime ’ which ensu es ha USB FIFOs
a e commi ed o he hos wi h in e als
o a mos 1.5ms, and on he hos side
he p ocessing in e als closely ma ched
his in e al.
IV. CONCLUSION
Con en ional machine ision using
ame-based senso s aces a undamen al
la ency-powe adeo . Low la ency can only be
achie ed by p ocessing a a high ame a e, which
bu ns mo e powe . The CPU load o less han 3%
achie ed in he slo ca ace is a esul o he low da a
a e a e aging 5keps ( housand e en s pe second) pe
ca . The s a ing came a scena io is ideal o using he
DVS, since only he small mo ing slo ca s c ea e
DVS e en s. The ea ly packe ime ansmi s
a ailable e en s om he came a a a minimum a e o
1/1.5ms=666Hz, which means ha mos packe s sen
o he hos con ain only abou 7 e en s pe ca . This is
a small amoun o da a o p ocess. By compa ison, i
he 240x180 pixel image could be ansmi ed o he
hos a 666Hz, i would mean a da a a e o 29M
pixels/s, which would be a ac o o abou 1000 imes
mo e da a.
The slo ca ace obo is a popula demons a ion
o he use o a DAVIS senso , mainly because acing
is un and i is a con es be ween human and compu e
ha in ol es quick eac ion imes. The p inciple o
ope a ion is simple and easy o explain. In p ac ice,
because he compu e is so p ecise, and because i can
use mo o b aking, i is p ac ically unbea able and so
o p oduce he illusion o a compe i i e ace i is
necessa y o enable he mode whe e he compu e ca
is slowed down i i is ahead.
The con ol o he ca is in some sense open-loop
because he h o le and b ake a e applied acco ding o
he ins an aneous posi ion o he ca on he ack,
ega dless o he ca ’s speed. Fu u e enhancemen s
could ocus on de eloping an adap i e model-based
con olle ha egula es he speed o he ca o a
desi ed le el ha is sa e o he cu a u e. Ou
a emp s o do his we e no success ul because he
model is su p isingly complex. The physics o ca
mo emen along he ack and he ac ion o he powe
applied o he ca on i s speed a e complica ed by
ack cu a u e, ic ion, indi idual ca a iabili y,
mo o hea ing, e c. Howe e he sho la ency o
senso measu emen could enable isual eedback on
h o le con ol.
Acknowledgemen s
This wo k was suppo ed by he Swiss NCCR Robo ics, EU
p ojec s SeeBe e (FP7-ICT-270324) and Visualise (FP7-ICT-
600954), Samsung and he DARPA SyNAPSE p ojec . P o o ypes
we e buil a he 2010 Tellu ide Neu omo phic Cogni ion
Enginee ing Wo kshop and he 2014 Capo Caccia Cogni i e
Neu omo phic Enginee ing Wo kshop. The implemen a ion
desc ibed he e was demons a ed a he 2014 Eu opean Compu e
Vison Con e ence.
Re e ences
[1] P. Lich s eine , C. Posch, and T. Delb ück, “A 128 x 128
120dB 15us La ency Asynch onous Tempo al Con as
Vision Senso ,” IEEE J Solid-S a e Ci cui s, ol. 43, no. 2,
pp. 566–576, 2008.
[2] C. B andli, R. Be ne , M. Yang, S.-C. Liu, and T. Delb uck,
“A 240x180 130 dB 3us La ency Global Shu e
Spa io empo al Vision Senso ,” IEEE J. Solid-S a e Ci cui s,
ol. Ea ly Access Online, 2014.
[3] A. Bolopion, Z. Ni, J. Agnus, R. Benosman, and S. Regnie ,
“S able hap ic eedback based on a dynamic ision senso o
mic o obo ics,” in 2012 IEEE/RSJ In e na ional Con e ence
on In elligen Robo s and Sys ems (IROS), 2012, pp. 3203–
3208.
[4] T. Delb uck and M. Lang, “Robo ic Goalie wi h 3ms
Reac ion Time a 4% CPU Load Using E en -Based
Dynamic Vision Senso ,” F on . Neu osci., ol. 7, p. 223,
No . 2013.
[5] jAER, jAER Open Sou ce P ojec . 2007 [Online]. A ailable:
h p://jae p ojec .o g
[6] “jAER / Code / [ 5522]
/jAER/ unk/s c/ch/unizh/ini/jae /p ojec s/ i ualslo ca .”
[Online]. A ailable:
h p://sou ce o ge.ne /p/jae /code/HEAD/ ee/jAER/ unk/s c
/ch/unizh/ini/jae /p ojec s/ i ualslo ca /. [Accessed: 25-Oc -
2014]
[7] “jAER / Code / [ 5522]
/de ices/pcbs/Slo Ca Con olle PCB.” [Online]. A ailable:
h p://sou ce o ge.ne /p/jae /code/HEAD/ ee/de ices/pcbs/S
lo Ca Con olle PCB/. [Accessed: 25-Oc -2014]
[8] slo Ca RacingTellu ide2010.wm . 2010 [Online]. A ailable:
h p://you u.be/ALneVn-Ls2Q?lis =UU70-
pVoJ4pwIAxqpgcR7oxg. [Accessed: 06-Oc -2014]
[9] Slo ca ace con olled by DVS Capo Caccia 2014. 2014
[Online]. A ailable: h p://you u.be/CnGPGiZuFRI.
[Accessed: 03-Oc -2014]
[10] Slo Ca Racing wi h DAVIS neu omo phic ision senso .
2014 [Online]. A ailable: h p://you u.be/AsO1TWS8_VA.
[Accessed: 28-Oc -2014]
Fig. 7. Slo ca posi ion and speed s. ime on a di e en ack. Two laps a e
comple ed success ully using mo o b aking o slow down jus a e ack e ex 0.
On he las lap, mo o b aking is disabled, esul ing in a c ash.