Expe imen s and Resul s
x.pan2@ u.nl
Pilo expe imen s: using Cha GPT o gene a e SPARQL que y o handc a ed ques ions wi h one-sho lea ning
Na u al language
ques ions
LLMs
SPARQL
que ies
Seman ic inaccu acies P oblem
Fail o link he co ec p ope ies and en i ies in ORKG
Wha is he maximum sample size?
Con ibu ion
E alua ion
Me ic
P34
P2006
P7046
S uc u al inconsis encies P oblem
Make e o s in que y s uc u e, such as missing o abundan links ( iples)
Wha a e he me ics used by pape "Using NMF-based ex summa iza ion
o imp o e supe ised and unsupe ised classi ica ion?
o kgp:P15687 d s:label Sample size (n) o kgp:P7101 d s:label has elemen s
Me hodology
Task de ini ion
Mo i a ion
FIRESPARQL: A LLM-BASED FRAMEWORK FOR SPARQL QUERY GENERATION OVER
SCHOLARLY KNOWLEDGE GRAPHS
Xueli Pan, Vic o de Boe , Jacco an Ossenb uggen
Da ase s
SciQA Benchma k, 100 handc a ed ques ions (HQs), 2465
au oma ically gene a ed ques ions (AQs).
Implemen a ion
LoRa ine- uning on Llama 3.2-3B Ins uc and Llama 3-8B
Ins uc , using 1,795 NLQ-SPARQL pai s om he SciQA
aining se .
The models we e ained unde a ious epoch
con igu a ions (3, 5, 7, 10, 15, and 20)
DeepSeek-R1-Dis ill-Llama-70B o he RAG
Qle e o SPARQL execu ion
a single NVIDIA H100GPU
Findings
Fine- uning signi ican ly imp o es accu acy
RAG alone does no gua an ee imp o emen — con ex
quali y ma e s
SPARQL Co ec o module imp o ed syn ac ic alidi y and
execu ion success.
HQs emain challenging, AQs a e easie because o
epe i i e empla es and consis en s uc u e.
B idging he gap be ween na u al language and
s uc u ed schola ly da a
Add essing he limi a ions o LLMs in schola ly
knowledge g aphs (SKGs) pa sing
Imp o ing accu acy o gene a ing SPARQL om
na u al language ques ions (NLQ)
Me ic
Con ibu ion
[1] Aue , Sö en, e al. "The sciqa scien i ic ques ion answe ing
benchma k o schola ly knowledge." Scien i ic Repo s 13.1
(2023): 7240.
