ELIA 3, 2002, pp. 33-48
CONTRASTIVE PERSPECTIVES ON THE RELIABILITY OF
GRAMMAR TESTING: THE CASE OF THE VERB PHRASE IN L2
ENGLISH
Gee Jacobs (Uni e si y o An we p)
Chis B aecke (Uni e si y o An we p)
In his a icle we p esen he p elimina y esul s o empi ical esea ch on he
eliabili y o g amma es ing. The ollowing es ing echniques a e
examined: mul iple choice, ansla ion, gap- illing and, inally, pe o mance.
In pa icula , o he pu poses o his pape we se up ou di e en es s o
iden i y knowledge o he English e b ph ase among na i e speake s o
Du ch. D awing om a de ailed examina ion o selec ed esul s i is
concluded ha a enewed in e es in a con as i e app oach o language
es ing and p ac ice is o be ecommended.
Key wo ds: g amma es ing, con as i e analysis, L2 English, he English
e b ph ase.
1. In oduc ion
This a icle epo s on he p elimina y esul s o an empi ically-based
in es iga ion in o selec ed g amma ical issues in ad anced-le el non-na i e
English. In pa icula , we se ou o examine he eliabili y o he a ious
echniques ha we used o iden i y knowledge o he English e b ph ase
among Du ch-speaking i s -yea unde g adua es a he Facul ies o Applied
Economics a wo Belgian (i.c. Flemish) uni e si ies. Fo ou pu poses,
eliabili y can be de ined as he consis ency o esul s om one es o
ano he (c . Bachman & Palme 1997).
Es udios de
lingüís ica inglesa aplicada
34 G. Jacobs and C.B aecke
ELIA 3, 2002, pp. 33-48
Wi hin he amewo k o e o analysis, he ollowing es ing
echniques will be deal wi h:
- mul iple choice
- ansla ion om Du ch (L1) in o English (L2)
- gap- illing (con ex ualized comple ion assignmen s)
- pe o mance: i.c. in o ma ion-gap and decision-making dyads
be ween es ees1.
The i s h ee echniques a e based on w i en asks and hey we e
used in di e en es s wi h di e en se s o be ween 400 and 500 es ees. As
a as he esul s a e conce ned, we will be p esen ing pe cen ages, some o
which a e based on ep esen a i e subco po a. No s a is ical analysis has
been ca ied ou . The ou h and inal es is based on o al communica ion:
we used audio- eco dings o dyads in ol ing some 35 es ees.
C ucially, he p esen a icle is mean o p o ide no mo e han a i s
explo a ion in o wha he p os and cons o he di e en echniques could be.
Tha ’s why, om he wide ange o da a a ailable, we will only ocus on he
ollowing ou speci ic ea u es o he g amma o he English e b ph ase,
all o which end o be p oblema ic o speake s o Du ch: iz.
- in o ma ion ques ions (o wh-ques ions)
- he pas and pas pa iciple in lec ions o i egula e bs
- he con inua i e use o he p esen pe ec
- absence o obliga ion
1 Egÿud & Glo e (2001) a gue ha such dyads lead lea ne s o p oduce highly a ied
language. In addi ion, Ga cia Mayo & Pica (2000) ha e shown ha hey a e no signi ican ly
di e en om dyads be ween lea ne s and na i e speake s.
Cons as i e pe spec i es on he… 35
ELIA 3, 2002, pp. 33-48
Fo all o hese ea u es o he g amma o he English e b ph ase,
he p oblems o na i e speake s o Du ch ha e been su icien ly desc ibed
(c . Zand oo ’s 1975 Handbook o English G amma as well as, mo e
ecen ly, Vandenbe gen, 1991 and Lemmens & Pa , 1995). The e o e, he
pu pose o ou es s, while diagnos ic, was no o make an in en o y o
p oblems bu a he o poin o a possible hie a chy o p oblems as well as
o mula ing en a i e hypo heses o e ining he desc ip ion.
2. In o ma ion ques ions
To s a wi h, we shall look a in o ma ion ques ions in which he ques ion
wo d is no he subjec , as in ansla ion-based mul iple choice es (1). Since
his is a pa o basic g amma , ew p oblems a e o be expec ed, excep
pe haps o he use o he ope a o , which is unknown in Du ch.
(1) Wa be eken da Engelse woo d?
a. Wha does ha English wo d mean? (82%)
b. *Wha means ha English wo d? (4%)
c. *Wha does ha English wo d means? (14%)
The esul s a e p edic able and su p ising a he same ime. P edicable,
because mos o he es ees op ed o a. Bu su p ising, because he majo
sou ce o mis akes was no he use o he ope a o , bu he in lec ion o he
main e b.
In e es ingly, ansla ion (2) – wi h no possible answe s sugges ed o
he es ees – yielded a he di e en esul s:
36 G. Jacobs and C.B aecke
ELIA 3, 2002, pp. 33-48
(2) Wa be eken da Engelse woo d waa o e je gis e en sp ak?
[Wha does ha English wo d mean you alked abou yes e day?]
Fi s o all, i should be poin ed ha one ou o six es ees came up wi h
a a he ee ansla ion, iz. a pa aph ase, in which he p oblem o he use o
he ope a o was a oided:
(2a) Wha is he meaning o ha English wo d you alked abou
yes e day? (17%)
O cou se, om a communica i e poin o iew his answe is pe ec ly
adequa e bu we should keep in mind ha i does poin o a i s es ic ion in
using ansla ion o es s uden s’ knowledge o speci ic i ems o g amma (c .
Ellis 1998 o an o e iew o esea ch on a oidance s a egies in language
lea ning and language es ing).
Le ’s e u n o (2) now. The in e es ing hing is ha his ime only ew
es ees came up wi h:
(2b) *Wha does ha English wo d means you alked abou yes e day?
(4%)
In con as , many mo e ailed o use he ope a o , which poin s o he
L1 ans e ha we we e expec ing:
(2c) *Wha means ha English wo d you alked abou
yes e day? (18%)
Indeed, i looks as i , by o e ing he igh al e na i e, mul iple choice
seems o unde es ima e he es ees’ eal p oblems (in his case he use o he
ope a o ). I s ole seems o be limi ed o ha o checking whe he es ees
ecognize he igh o m a he han es ing ac i e g amma knowledge. A
he same ime, he ques ion can be aised whe he p esen ing ung amma ical
al e na i es does no lead es ees o make mis akes ha hey would no
no mally make (-s o m).
Cons as i e pe spec i es on he… 37
ELIA 3, 2002, pp. 33-48
Al e na i ely, i emains o be seen i in (2) – as o he use o he
ope a o - L1 ans e was no induced by he ansla ion o ma . The esul s o
gap- illing es (3) seem o sugges ha his is no he case, since hey a e
almos iden ical o hose o he ansla ion es :
(3) Wha obse able u h _____ (CONTAIN, IT)?
(3a) *Wha obse able u h con ains i ? (14%)
(3b) *Wha obse able u h does i con ains? (4%)
Appa en ly, e en in he gap- illing assignmen , es ees seem o eso
o a ansla ing s a egy. This is in line wi h Ma a (1999), whose esea ch
ac ually e eals less mo he ongue in e e ence wi h ansla ion asks han
wi h gap- illing.
Le ’s e u n o he use o he ope a o . In e es ingly, in con as wi h he
esul s o in o ma ion ques ions, ou da a show ha he es ees don’ ha e a
p oblem wi h he use o he ope a o in simple yes-no ques ions, like “Do you
ag ee?” Ha dly anyone comes up wi h “Ag ee you?” Why his di e ence? We
belie e ha he answe may lie in he second ype o in o ma ion ques ions,
iz. hose in which he ques ion wo d is he subjec . Fo hese no ‘do’ is
equi ed, as in gap- illing es (4):
(4) Who _____ (BELIEVE) ha wa s can eally sol e humani a ian
p oblems?
S ill, qui e some es ees used do-suppo he e:
(4a) *Who does belie e ha wa s can eally sol e humani a ian
p oblems? (4%)
S angely, e en mo e used do-suppo in ansla ion es (5), whe e L1 ans e
should ha e led es ees away om i :

38 G. Jacobs and C.B aecke
ELIA 3, 2002, pp. 33-48
(5) Wa kos e mee ?
(5a) *Which does cos mo e? (7%)
These a e ela i ely small numbe s bu in ou iew i is su p ising ha
any mis akes we e made a all because, his ime, hey canno be ela ed o L1
ans e . Unless, o cou se, es ees mix up in o ma ion ques ions in which he
ques ion wo d is he subjec (and o which no do-suppo is used) wi h hose in
which he ques ion wo d is no he subjec (and o which do-suppo is used).
I his is he case, hen i seems o econ i m he need o a con as i e app oach
o selec ed issues o g amma and o cou se ansla ing should be pa o i , bo h
o p ac ice and es ing.
In he nex sec ion, we shall indica e ha a con as i e app oach need
no (and indeed should no ) be es ic ed o ansla ing.
3. Pas and pas pa iciple in lec ions o i egula e bs
In a ious gap- illing exe cises we es ed knowledge o he pas and pas
pa iciple in lec ions o i egula e bs. In he ollowing able i is shown
how many es ees made a mis ake o each o hem. In p esen ing he e bs
we ha e ollowed Qui k e al’s (1985) mo phological classi ica ion2.
2 Qui k e al (1985) use he ollowing h ee c i e ia:
- su ixa ion (i.e. he simple pas and pas pa iciple include a su ix)
- V-ed iden i y (i.e. he simple pas and pas pa iciple a e iden ical)
- Vowel iden i y (i.e. he in ini i e, he simple pas and he pas pa iciple show no
di e ence in base owel)
No e ha on he basis o hese c i e ia Qui k e al (1985) ha e se en classes. Fo he pu poses o
his a icle we ha e b ough Qui k e al’s i s and second classes oge he in ou i s class and we
ha e enumbe ed hei hi d class as ou second, hei ou h class as ou hi d, e c.
Cons as i e pe spec i es on he… 39
ELIA 3, 2002, pp. 33-48
Class 1: su ixa ion, V-ed iden i y, owel iden i y
BUILD 21%
LEND 44%
SEND 54%
Class 2: su ixa ion, V-ed iden i y
BUY 8%
FLEE 47%
KEEP 0%, 3%
MEAN 18%
SEEK 18%
Class 3: su ixa ion
BREAK 4%
DRIVE 5%
FORGET 0%
KNOW 5%
Class 4: V-ed iden i y, owel iden i y
SET 5%
Class 5: V-ed iden i y
GET 4%
LEAD 75%
MEET 1%
READ 6%
UNDERSTAND 4%
Class 6: /
BEGIN 12%
RING 4%
40 G. Jacobs and C.B aecke
ELIA 3, 2002, pp. 33-48
Appa en ly, class 1 is he only class ha poses majo p oblems
h oughou . Ce ainly – and pe haps su p isingly -, hose i egula e bs ha
ha e owel a ia ion (o so-called ‘ablau ’) seem a mo e simple, excep o
ela i ely unusual e bs like ‘seek’ and ‘ lee’. The high sco es o ‘mean’
and ‘lead’ can be ela ed o con usion abou spelling. Gene ally speaking,
hough, owel a ia ion seems ela i ely easy.
Bu why hen do he e bs in class 1 p o e so p oblema ic? We hink
he p oblem lies in de oicing he inal phoneme: while owel a ia ion is a
loud and clea indica o , one which speake s o Du ch also know in hei L1,
de oicing he inal phoneme is mo e sub le and comple ely un amilia o
hem. I ollows ha Du ch-speaking lea ne s o English should be made
awa e o he di e en classes o i egula e bs in gene al and o he oiced-
oiceless opposi ion o a numbe o he class 1 i egula e bs in pa icula .
The e a e wo in e es ing poin s o be made abou his con as i e issue: i
was e ealed by a gap- illing assignmen (and no by ansla ion) and i alls
beyond he each o ansla ion-based p ac ice.
4. The con inua i e use o he p esen pe ec
The con inua i e use o he p esen pe ec e e s o he use o he p esen
pe ec (simple o p og essi e) ‘ o in oke a s a e o a ai s ha began in he
pas , has con inued un il he speake ’s now and may be expec ed o endu e in o
he u u e’ (Mackenzie: 1997: 44). Du ch adi ionally uses a simple p esen
he e (Lemmens & Pa : 1995: 36). This con inua i e use o he English p esen
pe ec was es ed in he inal wo sen ences o ansla ion (6):
(6) Toen hij jong was, hee hij een jaa als kelne gewe k . Toen is hij
e huisd naa Londen. Daa s udee hij nu al sinds wee jaa aan de
uni e si ei . Hij hee e een nieuwe la sinds ap il.
[When he was young, he wo ked as a wai e o a yea . Then he
mo ed o London. The e he has s udied/been s udying a uni e si y o
wo yea s. He has had a new la since Ap il.]
Cons as i e pe spec i es on he… 41
ELIA 3, 2002, pp. 33-48
The esul s a e p edic able:
(6a) *The e he s udies a uni e si y o wo yea s. (42%)
(6b) *The e he is s udying a uni e si y o wo yea s. (24%)
(6c) *He has a new la since Ap il. (67%)
Again he ques ion can be aised whe he o no his ype o e o was
induced by he ansla ion o ma o he es . So le ’s also look a gap- illing
exe cise (7):
(7) Since Na o’s ai campaign agains Slobodan Milose ic go
unde way, Macedonia – a small, impo e ished s a e ha b oke away
om Yugosla ia in 1991 – ____ (FACE) a nigh ma e.
P esen pe ec 30%
Simple 15%
P og essi e 15%
P esen 49%
*Simple 17%
*P og essi e 32%
Pas 18%
He e, s ill a lo o s uden s op o he p esen . Jus like o
in o ma ion ques ions, i looks as i he e ec o ansla ion should no be
exagge a ed. Bu he e a e wo u he ema ks o be made.
Fi s , qui e some s uden s decided o selec a pas , which indica es
ha he ex ac was misunde s ood as ep esen ing a pas ime ame. This
seems o be a ecu ing p oblem wi h discou se-based gap- illing exe cises.
While con ex ualiza ion is mo e impo an o some g amma ical issues (e.g.
enses) and less so o o he s (like in lec ion o i egula plu als), i should be
poin ed ou ha i can also complica e ma e s.
48 G. Jacobs and C.B aecke
ELIA 3, 2002, pp. 33-48
Re e ences
Bachman, L.F. & Palme , A.S. 1997. Language Tes ing in P ac ice:
Designing and De eloping Use ul Language Tes s. Ox o d: OUP.
Be y, R. 2000. “’Youse - iendly’ Me alanguage: Wha E ec Does i Ha e
on Lea ne s o English?” IRAL 38: 195-211.
Együd, G. & Glo e , P. 2001. “O al Tes ing in Pai s”. ELT Jou nal 55: 70-
76.
Ellis, R. 1998. Second Language Acquisi ion. Ox o d: OUP.
Ga cia Mayo, M. & Pica, T. 2000. “L2 Lea ne In e ac ion in a
Fo eignLanguage Se ing: A e Lea ning Needs Add esssed?” IRAL
38: 35-58.
Lemmens, M. & Pa , T. 1995. Handboek oo de Ve ale Nede lands-
Engels. Leu en: Wol e s.
Mackenzie, J.L. 1997. P inciples and Pi alls o English G amma . Bussum:
Cou inho.
Ma a , H. 1999. “T ansla ion Elici a ion Techniques and Mo he -Tongue
In e e ence: Any Signi ican Connec ion?” IRAL 37: 307-320.
Nunan, D. 1998. “Teaching G amma in Con ex ”. ELT Jou nal 52: 101-109.
Pe o i z, W. 1997. “The Role o Con ex in he P esen a ion o G amma ”.
ELT Jou nal 51: 201-207.
S enson, N. 1991. “Induced E o s” in B.W. Robine & J. Schach e (eds.)
Second Language Lea ning: Con as i e Analysis, E o Analysis, and
Rela ed Aspec s. Ann A bo : The Uni e si y o Michigan P ess.
Vandenbe gen, A.-M. 1991. Explo ing English G amma . Leu en: Ga an .
Zand oo , R.W. 1975. A Handbook o English G amma . 7 h edn. London:
Longman.