F om da a o jou nalism
Nelly Ba e
4 h yea PhD s uden
Supe ised by Ioana Manolescu
In ia Saclay and Ins i u Poly echnique de Pa is
Janua y 31, 2024
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 1 / 47
Pe sonal (small) p esen a ion
My backg ound:
CS Bachelo @ Uni e si y o Lyon
CS Mas e , AI ack @ Uni e si y o Lyon
CS PhD s uden @ In ia and Ecole Poly echnique
My hesis is abou use -o ien ed explo a ion o semi-s uc u ed da a.
I is no only abou me:
...and many o he s!
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 2 / 47
Con ex : da a is he new gold (1/2)
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 3 / 47
Con ex : da a is he new gold (2/2)
Ou digi al wo ld comes wi h a ious con ex s, needs, ac o s, ...
We a e o e whelmed by ( aw) da a, we need o b ing o de
Ve y la ge and he e ogeneous da a:
Tables, ex , da abases, ...
De ec ion o en i ies o in e es :
People names, places, company names,
da es, ...
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 4 / 47
Da a + jou nalism = da a jou nalism
On one hand, we ha e:
Fac s
Da a
In he middle, we ha e:
Compu e s
P og ams
On he o he hand, we ha e:
Jou nalis s
Da a in es iga ion
Fac -checking
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 5 / 47
New use needs, especially in da a-jou nalism
Wi h he e ogeneous da a, use s need:
1Auni o m/in eg a ed iew o e he da a
2E icien and in ui i e ways o:
Ge a global unde s anding, desc ip ion o he da a
Ge in e es ing en i y connec ions
Que y and sea ch o in o ma ion in he sys em
3P oduce insigh s/ angible esul s o sha e
Bu :
They ha e ew o no CS skills
They do no know wha hey a e
exac ly looking o
Thei da a may be messy/di y
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 6 / 47
Vocabula y in oduc ion: da ase , schema, model
Da ase
A ile epo ing da a on a p ecise opic
Da a model
How da a is ep esen ed ( able, ex ,
da abase, ...)
Schema
How da a objec s a e designed and ela e
Da a he e ogenei y
A he model and/o schema le el
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 7 / 47
P oblem s a emen
e icien and exp essi e in eg a ion o he e ogeneous da a
=
P o ide a uni ied access (= pu in he same “box”)
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 8 / 47
P oblem s a emen
e icien and exp essi e in eg a ion o he e ogeneous da a
=
P o ide a uni ied access (= pu in he same “box”) o a se o da ase s
(wha e e hei p o enance, model, schema)
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 9 / 47
Connec ionS udio
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 16 / 47
Connec ionS udio p oblem s a emen
Ques ion: How o wo k wi h he e ogeneous (jou nalis ic) da a?
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 17 / 47
Connec ionS udio p oblem s a emen
Ques ion: How o wo k wi h he e ogeneous (jou nalis ic) da a?
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 17 / 47
Connec ionS udio p oblem s a emen
Ques ion: How o wo k wi h he e ogeneous (jou nalis ic) da a?
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 18 / 47
Connec ionS udio p oblem s a emen
Ques ion: How o wo k wi h he e ogeneous (jou nalis ic) da a?
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 18 / 47
Connec ionS udio solu ion in a nu shell
Ou answe : Ause - iendly da a lake o :
1Loading he e ogeneous da ase s
→uni o m/in eg a ed iew o e he da a
2S a is ics and da a summa ies
→global desc ip ion o he da a
3Keywo d sea ch
→que y and sea ch o in o ma ion
4Da a pa hs
→in e es ing en i y connec ions
5Que ying he da a lake
→que y and sea ch o in o ma ion
6Tabula -looking esul s
→p oduce insigh s/ angible esul s o sha e
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 19 / 47
Connec ionS udio solu ion in a nu shell
Ou answe : Ause - iendly da a lake o :
1Loading he e ogeneous da ase s
→uni o m/in eg a ed iew o e he da a
2S a is ics and da a summa ies
→global desc ip ion o he da a
3Keywo d sea ch
→que y and sea ch o in o ma ion
4Da a pa hs
→in e es ing en i y connec ions
5Que ying he da a lake
→que y and sea ch o in o ma ion
6Tabula -looking esul s
→p oduce insigh s/ angible esul s o sha e
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 19 / 47
Connec ionS udio solu ion in a nu shell
Ou answe : Ause - iendly da a lake o :
1Loading he e ogeneous da ase s
→uni o m/in eg a ed iew o e he da a
2S a is ics and da a summa ies
→global desc ip ion o he da a
3Keywo d sea ch
→que y and sea ch o in o ma ion
4Da a pa hs
→in e es ing en i y connec ions
5Que ying he da a lake
→que y and sea ch o in o ma ion
6Tabula -looking esul s
→p oduce insigh s/ angible esul s o sha e
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 19 / 47
Connec ionS udio solu ion in a nu shell
Ou answe : Ause - iendly da a lake o :
1Loading he e ogeneous da ase s
→uni o m/in eg a ed iew o e he da a
2S a is ics and da a summa ies
→global desc ip ion o he da a
3Keywo d sea ch
→que y and sea ch o in o ma ion
4Da a pa hs
→in e es ing en i y connec ions
5Que ying he da a lake
→que y and sea ch o in o ma ion
6Tabula -looking esul s
→p oduce insigh s/ angible esul s o sha e
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 19 / 47
Connec ionS udio solu ion in a nu shell
Ou answe : Ause - iendly da a lake o :
1Loading he e ogeneous da ase s
→uni o m/in eg a ed iew o e he da a
2S a is ics and da a summa ies
→global desc ip ion o he da a
3Keywo d sea ch
→que y and sea ch o in o ma ion
4Da a pa hs
→in e es ing en i y connec ions
5Que ying he da a lake
→que y and sea ch o in o ma ion
6Tabula -looking esul s
→p oduce insigh s/ angible esul s o sha e
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 19 / 47
1 Uni ied da a iew: a g aph
Ques ion: Wha i he da a g aph is huge?
Ou answe : Build i s compac ep esen a ion (summa y)
→We build a summa y g aph, wi h small in o ma ion loss
→E icien algo i hms and applica ions
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 22 / 47
1 Uni ied da a iew: a g aph
Ques ion: Wha i he da a g aph is huge?
Ou answe : Build i s compac ep esen a ion (summa y)
→We build a summa y g aph, wi h small in o ma ion loss
→E icien algo i hms and applica ions
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 22 / 47
1 Uni ied da a iew: a g aph
Ques ion: Wha i he da a g aph is huge?
Ou answe : Build i s compac ep esen a ion (summa y)
→We build a summa y g aph, wi h small in o ma ion loss
→E icien algo i hms and applica ions
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 22 / 47
2 S a is ics and da a summa ies
Ques ion: How o ge a quick o e iew o he da ase s?
Ou answe : Show Named En i ies s a s in cha s, ables and ag clouds
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 23 / 47
2 S a is ics and da a summa ies
Ques ion: How o ge a quick o e iew o he da ase s?
Ou answe : Show Named En i ies s a s in cha s, ables and ag clouds
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 23 / 47
2 S a is ics and da a summa ies
Ques ion: How o ge a quick o e iew o he da ase s?
Ou answe : Show Named En i ies s a s in cha s, ables and ag clouds
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 23 / 47
2 S a is ics and da a summa ies
Ques ion: How o ge a quick o e iew o he da ase s?
Ou answe : D aw an En i y-Rela ionship diag am ep esen ing
“impo an ” nodes
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 24 / 47
2 S a is ics and da a summa ies
Ques ion: How o ge a quick o e iew o he da ase s?
Ou answe : D aw an En i y-Rela ionship diag am ep esen ing
“impo an ” nodes
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 24 / 47
2 S a is ics and da a summa ies
Ques ion: How o ge a quick o e iew o he da ase s?
Ou answe : D aw an En i y-Rela ionship diag am ep esen ing
“impo an ” nodes
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 24 / 47
2 S a is ics and da a summa ies
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 25 / 47
5 Que ying he da a lake
Ou answe : Ge id o he “SQL w i ing pa ”, keep ables as ou pu
1Enume a e a se o pa hs in he summa y g aph:
p={n0,e0,n1,e1, ..., ei,ni+1}
2Each selec ed pa h pis associa ed o:
A sou ce a iable s( he i s elemen in p)
A a ge a iable ( he las elemen in p)
3Selec join p edica es (LEFT JOIN o INNER JOIN)
4Con e sion o a SQL que y:
Each pa h leads o a SQL que y, eusing sand
Pa h SQL que ies a e joined using join p edica es
p0={
s
z }| {
decla a ion, , gene al, , decla e , , name, ,
z }| {
# al}
./
p1={decla a ion
| {z }
s
, , inancialIn e es , , i ems, , i em
|{z}
}
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 28 / 47
5 Que ying he da a lake
Ou answe : Ge id o he “SQL w i ing pa ”, keep ables as ou pu
1Enume a e a se o pa hs in he summa y g aph:
p={n0,e0,n1,e1, ..., ei,ni+1}
2Each selec ed pa h pis associa ed o:
A sou ce a iable s( he i s elemen in p)
A a ge a iable ( he las elemen in p)
3Selec join p edica es (LEFT JOIN o INNER JOIN)
4Con e sion o a SQL que y:
Each pa h leads o a SQL que y, eusing sand
Pa h SQL que ies a e joined using join p edica es
p0={
s
z }| {
decla a ion, , gene al, , decla e , , name, ,
z }| {
# al}
./
p1={decla a ion
| {z }
s
, , inancialIn e es , , i ems, , i em
|{z}
}
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 28 / 47
6 Conc e e esul s
Ques ion: How o sha e esul s ound/c ea ed in he da a lake?
Ou answe : They can be expo ed (s a is ics, ables, que ies, ...)
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 29 / 47
6 Conc e e esul s
Ques ion: How o sha e esul s ound/c ea ed in he da a lake?
Ou answe : They can be expo ed (s a is ics, ables, que ies, ...)
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 29 / 47
6 Conc e e esul s
Ques ion: How o sha e esul s ound/c ea ed in he da a lake?
Ou answe : They can be expo ed (s a is ics, ables, que ies, ...)
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 29 / 47
S a Check
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 30 / 47
S a Check p oblem s a emen
Ques ion: How o ease/au oma e pa o he ac -checking p ocess?
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 31 / 47
S a Check p oblem s a emen
Ques ion: How o ease/au oma e pa o he ac -checking p ocess?
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 31 / 47
Connec ionS udio p oblem s a emen
Ques ion: How o ease/au oma e pa o he ac -checking p ocess?
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 32 / 47
Connec ionS udio p oblem s a emen
Ques ion: How o ease/au oma e pa o he ac -checking p ocess?
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 32 / 47
1 Consolida ed da a o F ench s a is ics
Ques ion: how o c ea e consolida ed da a o F ench s a is ics?
Ou answe : c awling o e e ence/ us ed websi es o F ench s a is ics
(Open Da a):
1INSEE: Ins i u Na ional de la S a is ique e des ´
E udes ´
Economiques
2Eu oS a : Eu opean s a is ical da abase
Knowing ha :
Da a is in di e en models ( ables, ex , ...)
Da a is huge
... and many o he conce ns ha we will no co e oday
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 34 / 47
1 Consolida ed da a o F ench s a is ics
Ques ion: how o c ea e consolida ed da a o F ench s a is ics?
Ou answe : c awling o e e ence/ us ed websi es o F ench s a is ics
(Open Da a):
1INSEE: Ins i u Na ional de la S a is ique e des ´
E udes ´
Economiques
2Eu oS a : Eu opean s a is ical da abase
Knowing ha :
Da a is in di e en models ( ables, ex , ...)
Da a is huge
... and many o he conce ns ha we will no co e oday
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 34 / 47
1 Consolida ed da a o F ench s a is ics
Ques ion: how o c ea e consolida ed da a o F ench s a is ics?
Ou answe : c awling o e e ence/ us ed websi es o F ench s a is ics
(Open Da a):
1INSEE: Ins i u Na ional de la S a is ique e des ´
E udes ´
Economiques
2Eu oS a : Eu opean s a is ical da abase
Knowing ha :
Da a is in di e en models ( ables, ex , ...)
Da a is huge
... and many o he conce ns ha we will no co e oday
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 34 / 47
1 Consolida ed da a o F ench s a is ics
Ini ial app oach: con e all da a in o a g aph
Yes, bu :
High cos o s o age (g aph size: 3Tb)
Sea ching he g aph was expensi e (1/3 o que ies we e e y long)
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 35 / 47
1 Consolida ed da a o F ench s a is ics
Ini ial app oach: con e all da a in o a g aph
Yes, bu :
High cos o s o age (g aph size: 3Tb)
Sea ching he g aph was expensi e (1/3 o que ies we e e y long)
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 35 / 47
1 Consolida ed da a o F ench s a is ics
Ou answe : con e da a ables as “a eas”
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 36 / 47
1 Consolida ed da a o F ench s a is ics
Ou answe : con e da a ables as “a eas”
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 36 / 47
1 Consolida ed da a o F ench s a is ics
Ou answe : con e da a ables as “a eas”
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 37 / 47
1 Consolida ed da a o F ench s a is ics
Ou answe : con e da a ables as “a eas”
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 37 / 47
1 Consolida ed da a o F ench s a is ics
Ques ion: How much mo e e icien is he no el app oach?
Ou answe :
Type INSEE Eu oS a To al
Files 96 207 7 094 103 301
Tables 112 966 7 003 119 969
A eas 1 286 603 12 179 533 13 488 136
G aph (Mb) 1 864 766 120 425 -
A eas (Mb) 577 8 055 -
Comp ession ×3 266 ×14 -
Lesson lea ned: he s o age should be chosen based on he da a/usage
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 38 / 47
2 Sea ching o s a is ics in he wa ehouse
Ques ion: How o e ie e in o ma ion om ex ual ques ions?
Ou answe : Gi en a que y Qcomposed o nkeywo ds kn:
Q= Taux
| {z }
k1
de chˆomage
| {z }
k2
2022
| {z }
k3
Find he 20 mos ele an ables, and possibly he alue:
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 39 / 47
3 Au oma ed ex analysis
Ques ion: Can we lea n hings om poli ical discou ses?
Ou answe : Yes, e.g., in wee s
Recognize/ex ac s a is ical claims
Iden i y well-known pe suasion echniques
Numbe o ollowed poli icians 63
Numbe o ga he ed wee s 77 081
Claims de ec ed (since 01/2022) 61 207
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 40 / 47
3 Au oma ed ex analysis
Ques ion: Can we lea n hings om poli ical discou ses?
Ou answe : Yes, e.g., in wee s
Recognize/ex ac s a is ical claims
Iden i y well-known pe suasion echniques
Numbe o ollowed poli icians 63
Numbe o ga he ed wee s 77 081
Claims de ec ed (since 01/2022) 61 207
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 40 / 47
3 Au oma ed ex analysis
Ques ion: Can we lea n hings om poli ical discou ses?
Ou answe : Yes, e.g., in wee s
Recognize/ex ac s a is ical claims
Iden i y well-known pe suasion echniques
Numbe o ollowed poli icians 63
Numbe o ga he ed wee s 77 081
Claims de ec ed (since 01/2022) 61 207
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 40 / 47
3 S a is ical claim de ec ion in poli ical discou ses
Ques ion: How o iden i y s a is ics used in poli ical discou ses?
Ou answe : Use a Machine Lea ning model ained on poli ical deba es
( om 1950 o 2024)
s a is ical claim = a s a is ical en i y + a alue + a da e
Taux de chˆomage en Guadeloupe
|{z }
s a is ical en i y
+ 27%
|{z}
alue
+ 2023
| {z }
da e
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 41 / 47
3 S a is ical claim de ec ion in poli ical discou ses
Ques ion: How o iden i y s a is ics used in poli ical discou ses?
Ou answe : Use a Machine Lea ning model ained on poli ical deba es
( om 1950 o 2024)
s a is ical claim = a s a is ical en i y + a alue + a da e
Taux de chˆomage en Guadeloupe
|{z }
s a is ical en i y
+ 27%
|{z}
alue
+ 2023
| {z }
da e
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 41 / 47
3 S a is ical claim de ec ion in poli ical discou ses
Ques ion: How o iden i y s a is ics used in poli ical discou ses?
Ou answe : Use a Machine Lea ning model ained on poli ical deba es
( om 1950 o 2024)
s a is ical claim = a s a is ical en i y + a alue + a da e
Taux de chˆomage en Guadeloupe
|{z }
s a is ical en i y
+ 27%
|{z}
alue
+ 2023
| {z }
da e
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 41 / 47
3 Pe suasion echniques de ec ion in poli ical discou ses
Ques ion: Can we de ec pe suasion echniques in poli ical discou ses?
Ou answe : Use ML classi ie s o de ec hose echniques and assign
hem a ca ego y
Emmanuel Mac on on No 28, 2023:
Le cap que je po e a oujou s ´e ´e le mˆeme:
| {z }
a i ma ion
´eindus ialise la F ance
| {z }
lag-wa ing
,gagne la ba aille du plein-emploi
| {z }
wa e m
,
ˆe e une Na ion plus sou e aine, indus ielle e d´eca bon´ee.
| {z }
lag-wa ing, hopes
Soyons des op imis es d´e e min´es.
| {z }
loaded language
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 42 / 47
3 Pe suasion echniques de ec ion in poli ical discou ses
Ques ion: Can we de ec pe suasion echniques in poli ical discou ses?
Ou answe : Use ML classi ie s o de ec hose echniques and assign
hem a ca ego y
Emmanuel Mac on on No 28, 2023:
Le cap que je po e a oujou s ´e ´e le mˆeme:
| {z }
a i ma ion
´eindus ialise la F ance
| {z }
lag-wa ing
,gagne la ba aille du plein-emploi
| {z }
wa e m
,
ˆe e une Na ion plus sou e aine, indus ielle e d´eca bon´ee.
| {z }
lag-wa ing, hopes
Soyons des op imis es d´e e min´es.
| {z }
loaded language
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 42 / 47
3 Pe suasion echniques de ec ion in poli ical discou ses
Ques ion: Can we de ec pe suasion echniques in poli ical discou ses?
Ou answe : Use ML classi ie s o de ec hose echniques and assign
hem a ca ego y
Emmanuel Mac on on No 28, 2023:
Le cap que je po e a oujou s ´e ´e le mˆeme:
| {z }
a i ma ion
´eindus ialise la F ance
| {z }
lag-wa ing
,gagne la ba aille du plein-emploi
| {z }
wa e m
,
ˆe e une Na ion plus sou e aine, indus ielle e d´eca bon´ee.
| {z }
lag-wa ing, hopes
Soyons des op imis es d´e e min´es.
| {z }
loaded language
Nelly Ba e (In ia) F om da a o jou nalism Janua y 31, 2024 42 / 47
