Documen a ion empla e o GenAI
Documen a ion empla e o he use
o Gene a i e AI in esea ch
Me ada a and decision-making c i e ia
By Thijs De iend , PhD – Science Policy O ice Open Science & Gene a i e AI
V ije Uni e si ei B ussel, Belgium
EUTOPIA-MORE T ain- he-T aine ma e ial
doi: 10.5281/zenodo.17243261
Documen a ion empla e o GenAI
1
Table o Con en s
Table o Con en s ...................................................................................................................................................................... 1
1. In oduc ion ............................................................................................................................................................... 2
2. When o use he empla e? ..................................................................................................................................... 2
3. Sec ions o he documen ......................................................................................................................................... 3
3.1 Explana ion ................................................................................................................................................................ 3
3.2 Decision-making c i e ia ............................................................................................................................................ 3
3.3 Me ada a .................................................................................................................................................................... 3
4. Examples o gene a i e AI applica ions .................................................................................................................... 5
4.1 Uploading long o ma i ems in gene a i e AI ......................................................................................................... 5
4.2 Assigning ca ego ies o esea ch pape s .................................................................................................................. 6
4.3 Conduc ing a sys ema ic e iew ............................................................................................................................... 6
5. Conclusion .................................................................................................................................................................. 6
Documen a ion empla e o GenAI
2
1. In oduc ion
Gene a i e AI is being inc easingly in eg a ed in o esea ch p ocesses. These include bu a e no limi ed o li e a u e
explo a ion, he gene a ion o code, and he execu ion o speci ic analyses (e.g., sen imen analysis). Despi e his
in eg a ion, he me ada a ela ed o use o gene a i e AI is no always being ully o co ec ly epo ed. Some esea ch
s udies omi en i ely essen ial de ails o wha models we e used o gene a e esponses. A common mis epo ed elemen
is he in e mixing o Cha GPT ( he se ice o e ed by OpenAI ha is buil on LLM echnology) and he unde lying base
models, such as GPT-4o. This leads o con using con o ions such as Cha GPT-4o. O he elemen s a e also o en missing,
which impac s he c edibili y o he esea ch and he deg ee o which esul s can be ep oduced.
A V ije Uni e si ei B ussels (VUB), we de eloped a s anda d o epo ing he use o gene a i e AI in esea ch p ocesses
(= he documen a ion empla e). The in en ion was o make VUB esea che s awa e o he ype o in o ma ion hey need
o epo when using gene a i e AI. The empla e was designed o be easy o use, and o be applicable o use o cha bo
se ices in a esea ch con ex . Fu he mo e, pa icula a en ion was paid o commonly mis epo ed elemen s in scien i ic
li e a u e.
2. When o use he empla e?
One ongoing deba e and commonly asked ques ion is when use o gene a i e AI is subs an ial enough o be epo ed in a
esea ch publica ion. While we canno p o ide de ini i e answe s o his ques ion ha would be applicable o all possible
scena ios ac oss all esea ch ields, we p o ide esea che s wi h h ee c i e ia o guide hei decision o epo use o
gene a i e AI. All hese c i e ia mus apply o he use o he empla e o be highly ecommended:
- Use o gene a i e AI mus be o a esea ch-speci ic pu pose (e.g., gene a ion o hypo heses) and no gene ic (e.g.,
making Powe poin -slides, p epa ing a speech).
- Use o gene a i e AI mus impac an in ellec ual s ep wi hin esea ch. This includes da a analysis, da a in e p e a ion,
design o an empi ical s udy, building a gumen a ion... The e m "in ellec ual" should be b oadly in e p e ed and
unde s ood wi hin he con ex o he esea ch domain. Fo ins ance, in esea ch ields ha do no adi ionally use
empi ical esea ch me hods and a e la gely ex -based (e.g., philosophy), easoning will be he p ima y in ellec ual s ep. A
simple example o a s ep ha is no conside ed in ellec ual is he w i ing down o s udy esul s in he Resul sec ion o he
pape .
- Use o gene a i e AI mus be mo e subs an ial han me ely p o iding mino assis ance o esea che -led asks. The use
mus pu he b un o he in ellec ual wo k o a speci ic esea ch p ocess on he model i sel a he han on he esea che .
Use wi hin a s uc u ed amewo k (e.g., as in ica e pa o a s udy design, sys ema ic li e a u e e iew, o o mal analysis
pipeline) is conside ed mo e subs an ial han ad hoc, uns uc u ed use (e.g., spo adic inqui ies o con enience).
We p o ide examples:
Wi hin scope (i in ellec ual s ep &
subs an ial use)
Ou side o scope
Sys ema ic hypo hesis gene a ion
G amma and s yle checking (modi ica ions a e
manually e iewed and no au oma ically inse ed)
Da a analysis au oma ion (e.g., sen imen & hema ic
analysis)
Li e a u e explo a ion (in o mal, no pa o
s uc u ed s udy)
Expe imen al design se up
Code debugging
Gene a ing a gumen s
Assis ing in li e a u e e iew design
Li e a u e syn hesis
Ci a ion o ma ing
Da a cleaning & ans o ma ion
C ea ing non- esea ch g aphs
De eloping in e iew ques ions
Gene a ing summa ies & abs ac s (uns uc u ed use,
no pa o me hod)
W i ing en i e pa ag aphs, including ci a ions
Answe ing ques ions based on pape
Documen a ion empla e o GenAI
3
3. Sec ions o he documen
3.1 Explana ion
The documen is spli in o wo sec ions. The i s sec ion includes in o ma ion (called “decision-making c i e ia”) ha may
help esea che s discuss whe he gene a i e AI should be employed. These may be epo ed wi hin he me hodology
sec ion. The second sec ion includes in o ma ion (called “me ada a”) ha should be epo ed as pa o he publica ion
i sel .
3.2 Decision-making c i e ia
In o ma ion
p o ec ion
#1
A e you aking measu es o be su e ha any da a a e going o be p ocessed
esponsibly and in acco dance wi h ele an legisla ion? You may wan o hink
abou (a) con ac ual a angemen s; (b) license condi ions; and (c) da a
p o ec ion egula ions applicable o you egion.
Con ex window
#2
A e you su e ha he p ocessing o in o ma ion you inpu going o ake place
wi hin he con ex window*o models? Please be awa e ha , e en wi hin he
con ex window, models do no always " ac o in" all in o ma ion o he same
deg ee o make p edic ions ("lead bias").
*Con ex window e e s o he maximum amoun o okens ha he model can
p ocess o o mula e an answe .
Ve i ica ion
#3
Wha measu es will you ake o e i y ou pu s? I you a e no aking measu es,
does empi ical e idence exis on accu acy o eliabili y o LLM use in simila
se ings?
Explainabili y
#4
I explainabili y* is ele an o you use o models, a e you going o ake s eps o
imp o e i ?
*Explainabili y e e s o he abili y o explain why a model p o ided a ce ain
esponse.
Accoun abili y
#5
A e you going o keep logs o all (o some) o you in e ac ions wi h he models?
T anspa ency
#6
Wha pa s o in e ac ions, i any, a e you going o epo wi hin you publica ion?
Plagia ism
#7
A e you going o ake s eps o ensu e p ope a ibu ion o ideas and sou ces
when engaging in idea ion? I so, how?
Resea ch
in eg i y
#8
Could he use o LLMs po en ially con lic wi h esea ch in eg i y (e.g., when using
LLMs o explo a o y da a analysis)?
D awbacks
#9
Could he d awbacks o he models unde mine he in ended pu pose o you
esea ch (e.g., in oducing bias, andomness p e en s ep oducibili y...)? I so,
how?
Ha ms
#10
Could he e po en ially be disp opo iona e socie al cos s associa ed wi h using
he models o hese pu poses (e.g., clima e impac , g oup ha m due o
biases...)?
Me hod
#11
Could he e be po en ial di e ences be ween ou pu s c ea ed by humans e sus
ou pu s c ea ed by models? I so, a e hese di e ences p oblema ic?
(e.g., hemes eme ging om hema ic analysis)
3.3 Me ada a
Ca ego ies
Ques ion
No.
Ques ion
De ini ions & Examples
Expec ed
o ma
Documen a ion empla e o GenAI
4
Basic
in o ma ion
#1
Wha se ice a e you using
(e.g., Cha GPT, Claude, Gemini,
API, so wa e o un models
locally…)?
Example: Cha GPT cha bo in e ace,
API o OpenAI models, models locally
un on my compu e …
Name o
se ice
#2
Wha model and model e sion
a e you accessing unde his
se ice?
Check he websi e o he
p o ide . I he e sion name is
no indable, epo he da e o
use.
Example: GPT-4o-2024-05-13, GPT-4-
1106-p e iew…
Model
e sion
Technical
cha ac e is ics
#3
A e you using an unmodi ied
model (i.e., base model) o a e
you wo king wi h a modi ied
e sion (e.g., ine- uned)?
Unmodi ied e sion = The model as i
was o iginally eleased (e.g., GPT-4,
Llama 3 8b)
Modi ied = The model as i was
o iginally eleased plus modi ica ions
ha we e made o he pa ame e s
(e.g., h ough ine- uning).
Fine- uned = Fu he adjus ing model
pa ame e s based on speci ic da ase s
o ailo models be e o one speci ic
ask
Ca ego y
#4
A e da a p ocessed locally o
elsewhe e?
Local p ocessing = The da a does no
lea e you compu e . The compu ing
powe comes om you RAM memo y
o a local sou ce .
No local p ocessing = The da a lea es
he compu e . Fo ins ance, i migh
be p ocessed on egional da a se e s
when you en e p omp s in o
cha bo s. A e wa d his, he da a will
also be s o ed o limi ed du a ion on
o he se e s, o en loca ed ou side
he Eu opean Union.
Ca ego y
#5
Wha is he cu -o da e o he
model?
Cu -o da e = The mos ecen da e
ha he aining da a has. Fo
ins ance, i he aining da a con ains
no da a mo e ecen han 1s o Ap il,
his will be he cu -o da e.
Da e
#6
Wha is he con ex window o
he model?
Con ex window = The maximum
amoun o okens ha he model can
p ocess o p edic he nex oken.
No e: This is only impo an o da a
analysis and long ex p ocessing (e.g.,
legal ex s, books…).
Numbe
Wo k low
#7
Wha ype o da a was inpu ed
(e.g., audio, abula da a,
p omp )?
Inpu da a = The da a you en e as a
use o he model. I ex , his is called
a "p omp ".
Inpu ype
#8
Wha ype o da a was
ou pu ed (e.g. ex , audio,
abula da a)?
Ou pu da a = The da a ha is
gene a ed. This may be na u al
language, images, code, ideo…
Ou pu ype

Documen a ion empla e o GenAI
5
#9
We e speci ic pa ame e s
manually se when wo king
wi h he model o no ? Only
answe o he inal
con igu a ion.
Pa ame e s = Va iables ha you may
gi e a ce ain alue when wo king wi h
he model (e.g., empe a u e, Top P...)
Ca ego y
#10
I e e , which pa ame e s did
you se manually? Only
desc ibe o he inal
con igu a ion.
Wha is he a ionale i you used
pa icula se ings o
pa ame e s? Desc ibe his in
you me hodology sec ion.
Pa ame e s,
Tex
#11
Did you euse exis ing p omp s
o use ce ain p omp ing
echniques (e.g., ew-sho
p omp ing, Chain-o -Though )?
I so, e e o hese in you
me hodology sec ion. Only
desc ibe he inal con igu a ion.
P omp s = An inpu , o en in na u al
language, ha elici s a esponse by he
model
P omp ing echniques = Techniques
which a e known h ough ei he
o mal esea ch o empi ical es ing o
elici mo e desi able ou pu s (e.g.,
highe quali y).
Ca ego y,
Re e ence
#12
How was he model applied and
in eg a ed wi hin you esea ch
wo k low? Desc ibe his in you
me hodology sec ion. Only
desc ibe he inal con igu a ion.
Me hod: Desc ibe he speci ic
p ocedu e, echnique, o app oach
used o apply he model. I a p e-
exis ing me hod was used, e e ence i
wi hin he me hodology sec ion.
Depic he p ocedu e in a lowcha .
Examples: Gene a ion ollowed by
manual e i ica ion, e ie al-
augmen ed gene a ion...
Ca ego y,
Re e ence
4. Examples o gene a i e AI applica ions
4.1 Uploading long o ma i ems in gene a i e AI
Imagine ha you upload an 800-page book on he his o y o in e na ional law. You would like o know wha speci ic
his o ical case law you could e e ence ela ed o speci ic cases ha you a e looking in o. A book has abou 300 wo ds pe
page, which o als abou 240.000 wo ds o he en i e book. This means he model has o p ocess abou 320.000 okens
(coun ing ha in gene al he amoun o okens mul iplied by 0.75 gi es he amoun o wo ds). Fo each case you p esen ,
you ge a lis o his o ical e en s.
He e is why epo ing documen a ion o gene a i e AI ma e s:
• Models ha e a maximum amoun o okens ha hey can p ocess be o e hey s a “ o ge ing” hings ( his
maximum is called he con ex window)
• I is known ha he e is al eady deg ada ion o pe o mance aking place e en wi hin he con ex window.
I you do no epo his in o ma ion, o he esea che s canno e i y whe he you use o he model made sense
me hodologically. I you op ed o a model wi h e y limi ed con ex window (e.g., ~200.000 okens), he model will no
be able o gi e answe s ha ake he en i e book in o accoun .
Documen a ion empla e o GenAI
6
4.2 Assigning ca ego ies o esea ch pape s
Imagine you a e doing a scien ome ic s udy. As pa o he me hodology, you a e using gene a i e AI o assign ca ego ies
o in o ma ion. (You don’ wan o ely on p e-exis ing classi ica ion schemes ha a e known o con ain many laws). You
a e assigning classi ie s o en i e pape s based on hei con en s. You decide o use one o OpenAI’s models ia hei API.
You lea e he hype pa ame e s in hei de aul alues as hey a e because you don’ unde s and hem.
He e is why epo ing documen a ion o gene a i e AI ma e s:
• The andomness o esponses is in luenced by se e al hype pa ame e s, including he empe a u e a iable.
The s anda d se ing migh be a alue be ween 0.8 and 1. A se ing 1, he model will occasionally pick
“unlikely” okens o he p edic ion. A se ing 0, he model will always pick he mos likely oken e e y ime,
leading o ep oducible ou pu s.
• The ca ego ies ha you assigned will be subjec o andomness! Doing he same un wice will esul in di e en
ca ego ies assigned.
• You jus used gene a i e AI as one s ep in an en i e wo k low. You need o documen and illus a e you en i e
wo k low in a lowcha .
I you do no epo his in o ma ion, o he esea che s canno e i y whe he you use o he model made sense
me hodologically. You assignmen o ca ego ies will be i ep oducible by o he esea che s i you jus use de aul alues
o hype pa ame e s. Addi ionally, he me hodological alidi y o you use o gene a i e AI will depend on he wo k low
i sel (i.e., does you use make sense).
4.3 Conduc ing a sys ema ic e iew
Imagine ha you a e conduc ing a sys ema ic e iew. You decide o use wo ways o iden i ying a icles. Fi s , you will
design sea ch s ings ha you un in academic da abases. You ead he abs ac s and con en s o he pape s. Based on
inclusion c i e ia, you hen decide whe he o include he a icles. Second, you decide o also ga he a icles ia Deep
Resea ch Modules o e ed by AI- ools. The Deep Resea ch Modules au ogene a e sea ch s ings and un hem agains
di e en academic da abases and sea ch engines. The Modules ead h ough he a icles and decide wha o include in
he inal epo . You e i y whe he hese a icles mee you inclusion c i e ia. I so, you ex ac he ci a ions and use hese
a icles in you e iew.
He e is why epo ing documen a ion o gene a i e AI ma e s:
• Using a Deep Resea ch Module is a comple ely di e en hing om jus using a s anda d La ge Language Model.
I is a wo k low ha includes mul iple s eps, such as sea ch s ing gene a ion, unning he sea ch s ings agains
academic da abases, eading h ough he collec ed a icles and d a ing an en i e epo on he opic. This
needs o be epo ed co ec ly.
• Deep Resea ch Modules canno do comp ehensi e sea ches as he e a e limi a ions o how many sea ch s ings
hey un, how many da abases hey can access and how many pape s hey ead. The numbe o un sea ch
s ings, accessed da abases and ead a icles mus also ideally be epo ed. A u he p oblem is ha he ou pu
o a Module is no ep oducible because he c ea ion o sea ch s ings gene a ed is subjec o andomness.
I you do no epo his in o ma ion, o he esea che s canno de e mine whe he you me hodology is sound. They may
also no unde s and he sho comings o he Deep Resea ch Module ha you used. I you ha e no e lec ed on he
limi a ions, you may w ongly po ay he Deep Resea ch Module as highly eliable and ep oducible.
5. Conclusion
This documen a ion empla e helps esea che s epo accu a e me ada a on he use o gene a i e AI in esea ch. This
boos s anspa ency o e he analy ical choices made, and ep oducibili y o he esea ch s udy. We encou age
esea che s o inco po a e he empla e in o hei wo k low (inso a no domain-speci ic s anda d al eady exis s). Jou nals
may also d aw inspi a ion om he epo ing s anda ds desc ibed in his documen .