id192425
DISCRETE GRAPH GENERATIVE MODELS FOR
SMALL MOLECULE GENERATION
LAURA GARCÍA ALFOCEA
Thesis supe iso
ALEXISMOLINAMARTINEZDELOSREYES(NOSTRUMBIODISCOVERYSL)
Tu o :ALFREDOVELLIDOALCACENA(Depa men o Compu e Science)
Deg ee
Mas e 'sDeg eeinA i icialIn elligence
Mas e 's hesis
School o Enginee ing
Uni e si a Ro i a i Vi gili (URV)
Facul y o Ma hema ics
Uni e si a de Ba celona (UB)
Ba celona School o In o ma ics (FIB)
Uni e si a Poli ècnica de Ca alunya (UPC) - Ba celonaTech
Abs ac
This p ojec in oduces Ca Mol, a gene a i e model designed o c ea e small molecules by
ep esen ing hem as g aphs, using node and edge ma ices wi h ca ego ical a ibu es.
A disc e e diffusion p obabilis ic model is applied o hese ep esen a ions, le e aging a
ma ginal dis ibu ion de i ed om he da ase , which imp o es pe o mance compa ed o
a uni o m dis ibu ion. To ensu e ha he model effec i ely lea ns he g aph dis ibu ion,
an a en ion mechanism inco po a ing edge in o ma ion is employed. The model is ained
on mul iple da ase s and e alua ed using a ious pe o mance me ics. On d ug-like
da ase s, Ca Mol demons a es compa able pe o mance o s a e-o - he-a models, wi h
supe io esul s in ce ain me ics. Addi ionally, he model’s pe o mance imp o es wi h
he size o he da ase , sugges ing scalabili y. Fu u e wo k will in ol e aining on e en
la ge da ase s o u he alida e i s scalabili y and po en ial.
Resumen
Es e p oyec o p esen a Ca Mol, un modelo gene a i o diseñado pa a c ea pequeñas
moléculas ep esen adas como g a os, u ilizando ma ices de nodos y a is as con a ibu os
ca egó icos. Se aplica un modelo p obabilís ico de di usión disc e a a es as ep esen a-
ciones, ap o echando una dis ibución ma ginal de i ada del conjun o de da os, lo que
mejo a el endimien o en compa ación con una dis ibución uni o me. Pa a asegu a que
el modelo ap enda co ec amen e la dis ibución del g a o, se emplea un mecanismo de
a ención que inco po a la in o mación de las a is as. El modelo se en ena con a ios con-
jun os de da os y se e alúa u ilizando di e en es mé icas de endimien o. En conjun os de
da os de moléculas simila es a á macos, Ca Mol demues a un endimien o compa able
con los modelos más a anzados, con esul ados supe io es en algunos casos. Además, el
endimien o del modelo mejo a con conjun os de da os más g andes, lo que sugie e su
escalabilidad. El abajo u u o inclui á en ena con conjun os de da os aún más g andes
pa a alida aún más su escalabilidad y po encial.
1
Resum
Aques p ojec e p esen a Ca Mol, un model gene a iu dissenya pe c ea pe i es
molècules ep esen ades com a g a os, u ili zan ma ius de nodes i a es es amb a ibu s
ca egò ics. S’aplica un model p obabilís ic de di usió disc e a a aques es ep esen acions,
ap ofi an una dis ibució ma ginal de i ada del conjun de dades, la qual millo a el endi-
men en compa ació amb una dis ibució uni o me. Pe assegu a que el model ap engui
co ec amen la dis ibució del g a o, s’emp a un mecanisme d’a enció que inco po a la
in o mació de les a es es. El model s’en ena amb di e sos conjun s de dades i es alo a
u ili zan di e en s mè iques de endimen . En conjun s de dades de molècules simila s a
à macs, Ca Mol demos a un endimen compa able amb els models més a ança s, amb
esul a s supe io s en alguns casos. A més, el endimen del model millo a amb conjun s
de dades més g ans, cosa que sugge eix la se a escalabili a . El eball u u inclou à
en ena amb conjun s de dades enca a més g ans pe alida més la se a escalabili a i
el seu po encial.
2
Acknowledgemen s
I would like o exp ess my deepes g a i ude o he en i e eam a Nos um Biodisco e y
o hei suppo h oughou he de elopmen o his p ojec . In pa icula , I would like
o offe my since es hanks o Alexis Molina and Ál a o Ciudad, whose guidance was
ins umen al in making his p ojec a eali y. Thei men o ship has been in aluable, and
I ha e lea ned so much om hem du ing he cou se o his wo k.
I would also like o hank Al edo Vellido o gi ing me his oppo uni y and o his
con inued suppo h oughou he p ojec , as well as my amily and colleagues, whose
unwa e ing suppo has been a cons an sou ce o s eng h and mo i a ion.
3
Con en s
Lis o Figu es 5
Lis o Tables 7
1 In oduc ion and Objec i es 8
2 S a e o he a 10
3 Me hods 12
3.1 G aphs...................................... 12
3.2 Diffusionmodels ................................ 14
3.2.1 Denoising Diffusion P obabilis ic Models (DDPMs) . . . . . . . . . 15
3.2.2 Disc e e Denoising Diffusion P obabilis ic Models (D3PM) . . . . . 16
3.2.3 Diffusion o g aphs . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.3 G aph Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.3.1 G aph ans o me . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.4 Me ics...................................... 22
4 Resul s 24
4.1 QM9da ase .................................. 24
4.2 D ug-likeda ase s................................ 25
4.3 Abla ions .................................... 30
5 Analysis o he sus ainabili y and e hical implica ions 32
6 Conclusions 33
Bibliog aphy 33
Appendix 37
A Da ase s..................................... 38
B Resul so heme ics.............................. 39
C Configu a ion .................................. 46
D Molecula gene a ion.............................. 47
E Examples o gene a ed molecules . . . . . . . . . . . . . . . . . . . . . . . 48
F Equi a iance................................... 49
4
Lis o Figu es
1 Molecula ep esen a ion o ace amide (CC(=O)N) ( ep esen ing he ca e-
go ies, no he encoded alues). . . . . . . . . . . . . . . . . . . . . . . . . 13
2 Diffusion p ocess o an image. . . . . . . . . . . . . . . . . . . . . . . . . . 14
3 Diffusion p ocess o a molecule. . . . . . . . . . . . . . . . . . . . . . . . . 15
4 T adi ional a en ion mechanism and i s modifica ion o add edge ea u e
in o ma ion.................................... 20
5 A chi ec u e o he G aph T ans o me model and a chi ec u e o each o
i slaye s. .................................... 22
6 Visualiza ion o he gene a ion p ocess o a alid molecule using ZINC250k
da ase . ..................................... 24
7 Compa a ion o he dis ibu ions o he gene a ed molecules aining he
model wi h QM9 da ase along wi h he dis ibu ions o he QM9 da ase . 26
8 Compa a ion o he dis ibu ions o he gene a ed molecules aining he
model wi h ZINC250k da ase along wi h he dis ibu ions o he ZINC250k
da ase . ..................................... 27
9 Compa a ion o he dis ibu ions o QED and LogP o he gene a ed molecules
aining he model wi h ZINC250k da ase along wi h he dis ibu ions o
heZINC250kda ase .............................. 28
10 Compa a ion o he dis ibu ions o SA and molecula weigh s o he gen-
e a ed molecules aining he model wi h ZINC250k da ase along wi h he
dis ibu ions o he ZINC250k da ase . . . . . . . . . . . . . . . . . . . . . 29
11 QED and LogP dis ibu ions o he gene a ed molecules wi h ZINC250k
da ase using ou model and Dig ess, along wi h he da ase dis ibu ions. 39
12 SA and molecula weigh dis ibu ions o he gene a ed molecules wi h
ZINC250k da ase using ou model and Dig ess, along wi h he da ase
dis ibu ions. .................................. 40
13 Compa a ion o he dis ibu ions o he gene a ed molecules aining he
model wi h MOSES da ase along wi h he dis ibu ions o he MOSES
da ase . ..................................... 41
14 Compa a ion o he dis ibu ions o QED and LogP o he gene a ed molecules
aining he model wi h MOSES da ase and along wi h he dis ibu ions
o heMOSESda ase .............................. 42
15 Compa a ion o he dis ibu ions o SA and molecula weigh s o he gen-
e a ed molecules aining he model wi h MOSES da ase and along wi h
he dis ibu ions o he MOSES da ase . . . . . . . . . . . . . . . . . . . . 43
16 QED and LogP dis ibu ions o he gene a ed molecules wi h a uni o m
dis ibu ion along wi h he dis ibu ions o he ZINC250k da ase . . . . . . 44
5
17 SA and molecula weigh s dis ibu ions o he gene a ed molecules wi h a
uni o m dis ibu ion along wi h he dis ibu ions o he ZINC250k da ase . 45
18 Examples o gene a ed alid molecules aining he model wi h he QM9
da ase . ..................................... 48
19 Examples o gene a ed alid molecules aining he model wi h he ZINC250k
da ase . ..................................... 48
20 Examples o gene a ed alid molecules aining he model wi h he MOSES
da ase . ..................................... 48
6
The e o e, a molecule can be ep esen ed by hese wo ma ices, Xand E, bo h con-
aining ca ego ical in o ma ion: he nodes and bond ypes. Rep esen ing hese ea u es
using one-ho encoding is pa icula ly ad an ageous in he gene a i e modeling p ocess.
This ep esen a ion enables he model o di ec ly p edic he p obabili ies o each node be-
longing o a specific a omic class and each edge co esponding o a specific bond ype. By
le e aging his p obabilis ic amewo k, he gene a i e model can effec i ely cap u e he
ca ego ical na u e o molecula g aphs and acili a e he gene a ion o ealis ic molecula
s uc u es.
Fo example, conside he molecule ace amide (C2H5NO), ep esen ed by he SMILES
s ing CC(=O)N. In his molecule, he e a e h ee a oms: ca bon (C), ni ogen (N), and
oxygen (O). The node ea u es, X, ep esen he a omic numbe s o each a om in he
molecule. The a omic numbe s o ca bon, ni ogen, and oxygen a e 6, 7, and 8, e-
spec i ely. The adjacency ma ix Eis a 4x4 ma ix encoding he bonding in o ma ion.
Specifically, he e is a single bond be ween he wo ca bon a oms (C-C), a single bond
be ween ca bon and ni ogen (C-N), and a double bond be ween ca bon and oxygen
(C=O). The ep esen a ion o his molecule, be o e applying one-ho encoding, can be
seen in Figu e 1.
(a) Ace amide molecule 2D
ep esen a ion using RDKi .
X=
6
6
8
7
(b) Node ea u es ma ix X.
E=
0100
1021
0200
0100
(c) Adjacency ma ix E.
Figu e 1: Molecula ep esen a ion o ace amide (CC(=O)N) ( ep esen ing he ca ego ies,
no he encoded alues).
Howe e , i is impo an o no e ha no e e y pai o ma ices Xand Eac ually
ep esen s a alid molecule. This is due o inhe en chemical cons ain s ha mus be
sa isfied o a molecula s uc u e o be chemically plausible, such as:
•Valency: Each a om has a specific alency, de e mined by i s elec on configu a ion,
which e e s o he numbe o bonds an a om can o m. Fo ins ance, ca bon can
ypically o m 4 bonds, oxygen 2 and hyd ogen jus 1.
•Bond ypes: Each ype o bond has diffe en p ope ies, such as he numbe o
sha ed elec ons, which mus be espec ed. Fo ins ance, in single bonds one pai
o elec ons is sha ed while in double bonds wo pai s o elec ons a e sha ed.
•No agmen a ion: A g aph ha ep esen s a molecule canno ha e disconnec ed
componen s unless i ’s a se o molecules.
These a e some o he in insic p ope ies ha he gene a i e model should lea n in
o de o gene a e ealis ic, chemically alid molecules. Ano he key p ope y o molecula
g aphs is ha he a oms (nodes) do no ha e a fixed o de . This means ha he s uc u e
o a molecule emains unchanged ega dless o how he nodes a e pe mu ed o eo de ed.
13
This pe mu a ion in a iance is a c ucial conside a ion when designing machine lea ning
models o molecules, as he model mus be able o p ocess molecula g aphs wi hou
elying on any a bi a y node o de ing.
3.2 Diffusion models
Gene a i e models a e a class o machine lea ning models designed o gene a e new da a
ha is simila o a gi en da ase by lea ning he unde lying dis ibu ion o he da a i sel .
Once ained, hese models can gene a e new, unseen samples ha esemble he da a
hey we e ained on, by jus sampling om his lea ned dis ibu ion. Many gene a i e
models, such as Va ia ional Au oencode s (VAEs) [27] o Gene a i e Ad e sa ial Ne wo ks
(GANs) [22], wo k by lea ning a la en space, i.e., a lowe -dimensional ep esen a ion o
he da a and hen gene a ing da a by sampling om his la en space and decoding o
ans o ming i back in o he o iginal da a space.
Ano he class o gene a i e models, diffusion models [28, 29], ope a e h ough a diffe -
en mechanism. These models a e cha ac e ized by he p og essi e in oduc ion o noise
in o da a and p og essi e emo al o ha noise. These p ocesses a e inspi ed by p inciples
o he modynamics, pa icula ly physical diffusion phenomena, such as he sp eading o
pa icles in a fluid, whe e ans o ma ions occu inc emen ally.
Denoising Diffusion P obabilis ic Models (DDPMs) [28] a e a specific subclass o di -
usion models ha use wo Ma ko chains. The o wa d p ocess adds noise o he da a
ollowing a pa ame e ized Ma ko chain, whe e he da a a each imes ep depends only on
he p e ious imes ep. Simila ly, he e e se p ocess is also modeled as a Ma ko chain,
p og essi ely denoising he da a in a s epwise manne .
The diffusion p ocess o an image can be seen in Figu e 2.
Figu e 2: Diffusion p ocess o an image.
In diffusion models designed o gene a ing new g aphs, he o wa d p ocess begins
wi h a g aph ha is g adually co up ed o e ime by in oducing noise, leading o a
14
loss o s uc u e and inc eased andomness. The model hen, du ing he e e se p ocess,
lea ns how o e e se his co up ion p ocess, eco e ing he o iginal g aph om i s noisy
e sion, effec i ely lea ning o gene a e new g aphs ha esemble he aining da a. These
wo p ocesses can be seen in Figu e 3.
Figu e 3: Diffusion p ocess o a molecule.
In his case, since nodes and edges a e disc e e ea u es ep esen ing he ype o edge
and he ype o bond, espec i ely, i is necessa y o use ca ego ical diffusion models.
These models ex end s anda d diffusion models o effec i ely handle disc e e o ca ego -
ical da a by p og essi ely in oducing noise in a manne ha p ese es he s uc u e o
ca ego ical a iables.
Disc e e Denoising Diffusion P obabilis ic Models (D3PM) [30] ep esen a pionee ing
amewo k wi hin his class o ca ego ical diffusion models. I models he diffusion p o-
cess wi h ansi ions be ween disc e e s a es using a ansi ion p obabili y ma ix, which
defines he p obabili y o ansi ioning be ween disc e e ca ego ies a each imes ep.
3.2.1 Denoising Diffusion P obabilis ic Models (DDPMs)
In DDPMs, he o wa d p ocess is modeled as a Gaussian noise-adding p ocess, so, o
any da a objec x, he Ma ko p ocess o adding noise is he ollowing, deno ing by x
i s one-ho encoding:
q(x |x −1) := N(x ;√1−β x −1, β I),(3.1)
whe e β is he a iance schedule. This leads o he ollowing closed- o m exp ession:
q(x |x0) = N(x ;√¯α x0,(1 −¯α )I),(3.2)
whe e ¯α := ∏
s=1 αs, being α := 1 −β .
As i can be seen om equa ion 3.1, he a iance schedule β1, ..., βTde e mines he
quan i y o noise ha is added in each imes ep. The e o e, i should be chosen o balance
15
he ade-off be ween sufficien ly des oying he o iginal da a o ensu e good gene a i e
pe o mance. We ha e implemen ed i using a cosine be a schedule [31].
A e adding noise o he da a du ing he o wa d p ocess, a neu al ne wo k is used
o lea n o emo e ha noise and eco e back he o iginal molecule. This model akes
as inpu he noisy da a x a a gi en imes ep and he imes ep i sel and p edic s he
logi s o he clean o iginal da a pθ(x|x ). The imes ep in o ma ion is added o he model
by using sinusoidal posi ional embeddings [32], which p o ide a unique ep esen a ion o
each posi ion in he sequence. The op imiza ion is done by using he a ia ional lowe
bound on he nega i e log likelihood, which can be w i en as he sum o h ee diffe en
e ms:
L=Lp io +Ldi −Eq[log pθ(x|x1)] ,(3.3)
wi h
Lp io =DKL (q(xT|x)∥p(xT)) ,(3.4)
and
Ldi =
T
∑
=1
DKL (q(x −1|x ,x)∥pθ(x −1|x )) ,(3.5)
whe e DKL ep esen s he Kullback-Leible di e gence be ween wo dis ibu ions and
p(xT=N(xT;0,I)) ep esen s he limi dis ibu ion, a Gaussian dis ibu ion in he case
o DDPMs.
The e e se p ocess is also modeled by a Ma ko chain using he lea ned Gaussian
ansi ions and s a ing om he limi dis ibu ion. The whole chain can be w i en as:
pθ(x0:T) = p(xT)
T
∏
=1
pθ(x −1|x ),(3.6)
while each s ep ollows:
pθ(x −1|x ) = N(x −1;µθ(x , ),Σθ(x , )),(3.7)
whe e Σθis usually fixed o simplici y and he exp ession o µcan be ob ained using
he p edic ion o he model. The denoised da a is ob ained by applying hese o mula
g adually o all he imes eps.
3.2.2 Disc e e Denoising Diffusion P obabilis ic Models (D3PM)
D3PMs a e a gene aliza ion o DDPMs o ca ego ical da a, by going beyond co up ion
p ocesses o p ocesses wi h diffe en possible ansi ion ma ices [33].
Conside ing a ca ego ical a iable ha can ake on Kdiffe en ca ego ies, i.e., x , x −1∈
{1, . . . , K}, he o wa d p ocess is modeled using ansi ion p obabili y ma ices Q , whe e
[Q ]ij =q(x =j|x −1=i)specifies he p obabili y o ansi ioning om ca ego y i o
ca ego y ja imes ep .
Wi h hese ansi ion ma ices, he p obabili y o x gi en x −1, ep esen ing by x
and x −1 hei one-ho encoding, is exp essed as:
q(x |x −1) = Ca (p=x −1Q ),(3.8)
16
which is a ca ego ical dis ibu ion wi h p obabili ies gi en by he ow ec o p. The e o e,
q(x |x0) = Ca (x ;p=x0Q ),wi h Q =Q1Q2. . . Q .(3.9)
I can be seen ha , unlike con inuous diffusion, ca ego ical diffusion models allow
o a a ie y o ansi ion p obabili y ma ices h ough he design o Q , which defines
he limi dis ibu ion. This limi dis ibu ion co esponds o he p obabili y dis ibu ion
o g aphs ha ep esen pu e noise. Some o he possible ansi ion ma ices a e he
ollowing ones:
•Uni o m: The ansi ion p obabili y o e e y s a e is uni o m: Q = (1 −β )I+
β
K1K×K. In his case, cumula i e p oduc s can be compu ed in close o m Q =
α I+β
K1K×K.The limi dis ibu ion is he e o e uni o m.
•Ma ginal: The ma ginal ansi ion ma ix ep esen s he p obabili ies o he da ase
i sel , ensu ing ha he ansi ions espec he inhe en s uc u e o dis ibu ion o
he da ase . In his case, he ma ix is designed based on he obse ed equencies
o diffe en ca ego ies in he da a: Q = (1−β )I+β 1KmT, whe e m ep esen s he
ma ginal dis ibu ion o he da ase . The cumula i e p oduc also has a closed o m
in his case: Q =α I+β 1KmT. The limi dis ibu ion in his case co esponds
o he dis ibu ion o he da ase .
I has been es ed ha conside ing he ma ginal dis ibu ions o he da ase can speed
up bo h he aining and sampling p ocesses. This is because noised g aphs a e close o
eal g aphs han in he case o uni o m noise, in which noised g aphs a e eally andom
and he e o e e y a om ep esen ing eal molecules. The e o e, we conside his limi
dis ibu ion, which is diffe en o each o he da ase s.
In he case o ca ego ical diffusion, he noisy da a is sampled om he limi dis ibu-
ion and he sequence o g aphs ob ained du ing he denoising p ocess can be ob ained
using he ollowing o mula:
pθ(x −1|x )∝∑
x0
q(x −1,x |x0)pθ(x0|x ),(3.10)
whe e and x0is he p edic ed clean da a ob ained om he ained model. The e e se
p ocess consis s in applying his o mula o each o he imes eps un il ob aining he final
comple ely denoised da a x0.
The ac able exp ession o compu e he e e se p ocess is he ollowing:
q(x −1|x ,x0) = q(x |x −1,x0)q(x −1|x0)
q(x |x0)=Ca (x −1;p=x Q⊤
⊙x0Q −1
x0Q x⊤
)(3.11)
whe e ⊙deno es elemen -wise mul iplica ion.
3.2.3 Diffusion o g aphs
In he case o conside ing molecules ep esen ed as g aphs, noise can be added inde-
penden ly o nodes and edges [3] and, since hey bo h a e ca ego ical da a, he e a e
wo ansi ion ma ices QX
and QE
and he ca ego ical dis ibu ion o he noisy da a is
de e mined by he ollowing p obabili ies:
17
q(G |G) = Ca (XQX
,EGE
),(3.12)
wi h
QX
=QX
1QX
2. . . QX
and QE
=QE
1QE
2. . . QE
,(3.13)
all o hem ob ained using he ma ginal dis ibu ions o nodes and edges om he da ase ,
as explained be o e.
Once his compu a ion is done, he noisy g aph a he pa icula imes ep ,G =
(X ,E ), is ob ained by sampling he nodes and edges om he ob ained dis ibu ions.
Rega ding he e e se p ocess, he noisy g aph, along wi h hei numbe o nodes, a e
sampled om he limi dis ibu ion, GT∼qX×qE. This g aphs is p og essi ely denoised,
ob aining he sequence o g aphs GT−1, ..., G1, un il ob aining he g aph wi hou noise,
G0.
The sequence o noisy g aphs can be ob ained using equa ions 3.10 and 3.11 indepen-
den ly o nodes Xand edges E. Wi h hese equa ions, he g aph is denoised s ep by s ep
o he T imes eps un il he final g aph G0is ob ained.
A e adding noise o he da a du ing he o wa d p ocess, a neu al ne wo k is used
o lea n o emo e ha noise and eco e back he o iginal molecule. This model akes
as inpu he noisy g aph G = (X ,E )a a gi en imes ep and he imes ep embedding
and p edic s he logi s o he clean o iginal da a pθ(G|G ).
The op imiza ion o he pa ame e s o he model o p edic he o iginal molecule could
be done by op imizing he a ia ional lowe bound on he nega i e log likelihood, which
can be w i en as ollows:
L=Lp io +Ldi −Eq[log pθ(G|G1)] ,(3.14)
wi h
Lp io =DKL (q(GT|G)∥qX×qE),(3.15)
and
Ldi =
T
∑
=1
DKL (q(G −1|G , G)) ∥pθ(G −1|G ).(3.16)
The e ms qXand qEa e he ma ginal dis ibu ions o nodes and edges o he da ase
and pθ(G|G1)a e he p obabili ies o he clean g aph gi en he las noisy g aph G1. Each
o he desc ibed e ms ha e o be compu ed independen ly o he nodes enso Xand
o he edges enso E.
Howe e , ecen diffusion models a e op imized using diffe en objec i es. We only
op imize he las e m o he lowe bound. The fi s e m (equa ion 3.14) in he case
o ca ego ical diffusion is always ze o since bo h dis ibu ions a e he limi dis ibu ions.
The second e m (equa ion 3.15) was e y small and has no been conside ed in ou model
because o nume ical s abili y (Sec ion 4.3).
The e o e, he only e m le is he las one, whose expec ed alue ep esen s he c oss
en opy. The e o e, he model has been ained by op imizing he c oss en opy be ween
he p obabili ies o he p edic ed node and adjacency ma ix enso s and he o iginal
ones.
18
3.3 G aph Neu al Ne wo ks
F om he ep esen a ion o molecules as g aphs explained in Sec ion 3.1, wo key aspec s
can be highligh ed.
The fi s one is he s uc u e. In he case o images, da a is ep esen ed on a egula
g id whe e each pixel has a fixed numbe o neighbo s, as e e y poin is uni o mly con-
nec ed o i s immedia e su oundings. In con as , in g aphs, he numbe o neighbo s
o each node is de e mined by he numbe o edges i possesses, leading o a a ying
numbe o neighbo s o diffe en nodes. Fu he mo e, du ing he molecule gene a ion
p ocess, edges a e con inuously added and emo ed, esul ing in a dynamic and i egula
s uc u e. This i egula i y in oduces challenges o he di ec applica ion o adi ional
con olu ional ope a ions.
The second aspec conce ns he o de ing o nodes. A molecule wi h a eo de ing o
he nodes will ha e a diffe en nodes enso and adjacency ma ix; howe e , he molecule
i ep esen s emains he same. The e o e, he model’s ou pu should emain consis en
o bo h eo de ed molecules.
To add ess hese challenges, G aph Neu al Ne wo ks (GNNs) [34] ha e been de eloped
as specialized a chi ec u es capable o effec i ely p ocessing and lea ning om he dynamic
and i egula na u e o g aph-s uc u ed da a, while also p ese ing node equi a iance.
Node equi a iance ensu es ha he model’s p edic ions emain consis en ega dless o
he node o de ing o pe mu a ion. This p ope y is c ucial o cap u ing he inhe en
symme y in g aph da a and imp o ing gene aliza ion ac oss a ious g aph s uc u es.
GNNs a e a class o machine lea ning models specifically designed o wo k wi h g aph-
s uc u ed da a. A each laye o he GNN, a node agg ega es in o ma ion om i s
neighbo s and upda es i s own ep esen a ion. This p ocess enables he ne wo k o cap u e
how he su ounding con ex influences a nodes s a e, which is specially use ul in molecula
modeling, whe e he p ope ies o an a om can be hea ily influenced by he a oms i is
bonded o.
The key idea behind GNNs is message passing [35]. Du ing he o wa d pass, each
node sends in o ma ion (messages) o i s neighbo s along he edges o he g aph. These
messages a e agg ega ed, usually by some o m o summa ion, a e aging, o conca ena ion
and used o upda e he node’s ea u e ep esen a ion. This p ocess allows he GNN o
lea n how he su ounding con ex influences a node’s s a e. A e he message passing
and node upda es, a eadou unc ion is o en used o agg ega e in o ma ion ac oss all
nodes in he g aph o p oduce a global ep esen a ion.
3.3.1 G aph ans o me
The sel -a en ion mechanism used in T ans o me models [32] can be iewed as a flexible
and dynamic way o agg ega e in o ma ion in GNNs. This sel -a en ion mechanism allows
o p ocess all okens in a sequence simul aneously, enabling highly efficien pa alleliza ion
and he abili y o cap u e long- ange dependencies.
I is ob ained by compu ing a se o a en ion sco es ha allow each oken o ocus on
e e y o he oken in he sequence using he Que y (Q), Key (K) and Value (V) ec o s.
The sel -a en ion sco es a e gi en by:
19
(a) A en ion mechanism.
(b) A en ion mechanism modified us-
ing edge ea u es (E).
Figu e 4: T adi ional a en ion mechanism and i s modifica ion o add edge ea u e in-
o ma ion.
A en ion(Q, K, V ) = so max(QKT
√dk)V. (3.17)
We also apply no maliza ion laye s o he in e media e ep esen a ions o he a en ion
sco es, due o he ac ha a en ion logi s can g ow uncon ollably, leading o almos
de e minis ic (one-ho ) a en ion dis ibu ions, causing anishing g adien s [36]. This
no maliza ion has imp o ed ou aining s abili y.
The o iginal ans o me a chi ec u e was designed o Na u al Language P ocessing
(NLP) asks, whe e i ope a es on sequences o wo ds ha a e implici ly ep esen ed
as ully connec ed g aphs, wi h connec ions be ween all wo ds. Howe e , his app oach
pe o ms poo ly when no e e y wo pai s o nodes a e connec ed. Because o his, he
sel -a en ion mechanism should also ake in o accoun he in o ma ion abou whe he
wo nodes a e connec ed o no and how hey a e connec ed. This is done by mul iplying
he a en ion sco es by hese ea u es using an elemen -wise p oduc , allowing he e o e
he a en ion mechanism o ake in o accoun he connec i i y o he nodes.
The a en ion mechanism inco po a ing edge ea u e in o ma ion is illus a ed in Fig-
u e 4. This figu e p esen s a schema ic ep esen a ion o he a en ion mechanism: one
e sion (Figu e 4a) shows he s anda d o mula ion using only he ma ices Q, K and V
de i ed om he nodes enso (X), while he o he (Figu e 4b) includes he addi ional
edge ea u e in o ma ion (E) as p oposed in [37]. In bo h cases, he ou pu consis s o
he a en ion sco es.
The g aph ans o me a chi ec u e [37] was designed o add ess his limi a ion by
adap ing he ans o me a chi ec u e o g aph-s uc u ed da a. This is done by, fi s
o all, inco po a ing edge ea u es in o ma ion o he a en ion mechanism as explained
be o e, enabling he model o cap u e he ela ionships be ween nodes based on he g aph
s uc u e. Mo eo e , he posi ional encodings a e modified o be e eflec he complex
opology o g aphs.
Gi en ha g aphs do no ha e a na u al o de (like sequences) and ha ans o me
models p ocess he en i e inpu sequence in pa allel, one key challenge is he inabili y
20
o model sequen ial o de . To add ess his, posi ional encodings PE a e in oduced o
p o ide in o ma ion abou he ela i e o absolu e posi ion o okens in he sequence.
Howe e , as discussed be o e, he e is no single "co ec " way o assign unique posi ions o
nodes. Because o his, posi ional encodings on g aphs should be based on hei s uc u al
p ope ies.
One common ype o PE used o g aphs a e Random Walk PE [38]. These a e
compu ed using a andom walk diffusion p ocess and le e age he concep o andom
walks, which a e pa hs h ough a g aph ha mo e om one node o ano he based on he
g aph’s s uc u e. The idea is o calcula e he p obabili y ha a andom walk s a ing a
node iwill e u n o node ia e ks eps, which a e ep esen ed as:
pRW P E
i= [RWii, RW 2
ii, . . . , RWk
ii]∈Rk,(3.18)
whe e RWii deno es he sel - e u n p obabili y o node iin a andom walk o e u n
o i sel and kis he numbe o andom walk s eps conside ed. I is ob ained using he
andom walk ma ix RW =AD−1, whe e Ais he adjacency ma ix and D he deg ee
ma ix.
We use andom walks as posi ional encodings. They a e added o he inpu da a
du ing aining as lea nable pa ame e s, using a d opou ac o ha andomly u ns
hem o ze o, allowing he model o ope a e wi hou elying on hem du ing in e ence
[38].
The backbone o he model is illus a ed in Figu e 5a, whe e he inpu consis s o he
node enso X, augmen ed wi h andom walks as posi ional encodings (PE) and imes ep
embeddings, as well as he edge ea u e enso E. These inpu s a e p ocessed h ough
Feed Fo wa d Ne wo ks (FFN), which se e o embed he addi ional in o ma ion in o he
model. A se ies o laye s is hen applied, ollowed by ano he FFN o p oduce he final
ou pu .
The s uc u e o each laye is de ailed in Figu e 5b. As shown, he p ocess inside
each o he laye s begins by applying he sel -a en ion mechanism ac oss he heads. The
a en ion sco es om each head a e hen conca ena ed, ollowed by he applica ion o a
se ies o linea laye s, along wi h some addi ion and no maliza ion laye s o efine he
node and edge ep esen a ions and gene a e he final ou pu .
21
(a) Backbone o he G aph T ans-
o me model.
(b) Each laye o he G aph T ans-
o me model.
Figu e 5: A chi ec u e o he G aph T ans o me model and a chi ec u e o each o i s
laye s.
3.4 Me ics
We will explain some o he mos ele an me ics in 2D molecule gene a ion, p oposed by
[4]. I is impo an o no e ha while all hese me ics assess he quali y o he gene a ed
se , he ele ance o each me ic depends on he specific ask he model is designed o .
Op imizing one me ic may come a he expense o ano he , so he choice o me ics
should align wi h he desi ed applica ion.
•Validi y: Measu es he p opo ion o gene a ed molecules ha a e chemically alid,
meaning hey sa is y essen ial chemical cons ain s. A high alidi y sco e indica es
he model has lea ned he da ase ’s unde lying dis ibu ion and in insic p ope ies,
while a low sco e sugges s he model gene a es in alid molecules and ails o cap u e
he da a dis ibu ion p ope ly.
•Uniqueness: Reflec s he p opo ion o unique molecules gene a ed, a e emo ing
duplica es. A high uniqueness sco e sugges s he model can gene a e a di e se se
o new molecules, whe eas a low sco e indica es model collapse, whe e he model
epea edly gene a es simila o iden ical molecules.
•No el y: Measu es he p opo ion o alid gene a ed molecules no p esen in he
aining se . A low no el y sco e indica es o e fi ing, whe e he model eplica es
he aining da a ins ead o gene a ing new, unseen samples.
•F éche ChemNe Dis ance (FCD): Compa es he dis ibu ion o gene a ed
molecules wi h he aining da ase using molecula embeddings [39]. These em-
beddings a e de i ed om a neu al ne wo k ained o p edic biological p ope ies.
FCD unifies he alidi y, chemical and biological ele ance o he molecules and
di e si y.
I is impo an o no e ha while he F eche ChemNe Dis ance (FCD) is a eliable
me ic o measu ing he simila i y be ween he gene a ed dis ibu ion and he da ase
dis ibu ion, i elies on embeddings de i ed om he ac i a ions o a neu al ne wo k
22
(a) SA dis ibu ions.
(b) Molecula weigh s dis ibu ions.
Figu e 10: Compa a ion o he dis ibu ions o SA and molecula weigh s o he gene a ed
molecules aining he model wi h ZINC250k da ase along wi h he dis ibu ions o he
ZINC250k da ase .
Me ic Ca Mol Dig ess VAE
Validi y 73.47% 85.7% 97.7%
Uniqueness 99.39% 100.0% 99.8%
No el y 99.26% 95.0% 69.5%
FCD 4.29 1.19 0.57
Table 3: Compa ison o he e alua ion me ics ob ained o he MOSES da ase using
Ca Mol, Dig ess and a VAE. The esul s ha e been ob ained om [3].
The esul s ob ained on he MOSES da ase su pass hose achie ed on ZINC250k.
This indica es ha as he size and di e si y o he da ase inc ease, he model is be e
able o lea n he unde lying da a dis ibu ions and gene a e molecules ha mo e closely
adhe e o hese pa e ns. A la ge da ase p o ides mo e aining samples and g ea e
a iabili y, which enhances he model’s capaci y o gene alize, esul ing in highe alidi y
and no el y sco es.
These findings indica e ha he model’s pe o mance is highly dependen on he
quali y and di e si y o he aining da ase . To u he alida e his, he model will
be es ed on a significan ly bigge subse o he ZINC da ase , which encompasses he
MOSES and ZINC250k da ase s along wi h many addi ional samples. This da ase is
29
significan ly la ge and mo e di e se, p o iding a iche ounda ion o aining. This
nex s ep aims o p o e ha inc easing he da ase ’s size and complexi y can u he
imp o e he models capaci y o gene a e alid and no el molecules ha closely esemble
eal-wo ld molecula dis ibu ions.
4.3 Abla ions
To demons a e he impo ance o including posi ional encodings as pa o he inpu o
he gene a i e model, we conduc ed addi ional expe imen s using he ZINC250k da ase .
Specifically, we ained he model wi hou any posi ional encodings and adding hem o
he inpu o all he molecules in he da ase . The esul s ob ained in bo h cases cases
we e significan ly wo se han he esul s ob ained wi h ou s anda d se up whe e andom
walk posi ional encodings a e added o he inpu wi h a d opou ac o o 0.7.
Wi hou posi ional encodings, he gene a ed a om and bond dis ibu ions emained
closely aligned wi h hose o he da ase . Howe e , he alidi y a e d opped o 46.73%,
wi h almos all gene a ed molecules being no el and unique. This sugges s ha while he
model main ains di e si y, i s uggles o gene a e a high p opo ion o alid molecules
wi hou posi ional encodings. Addi ionally, he FCD me ic was measu ed a 12.19, indi-
ca ing ha he gene a ed molecules we e less simila o hose in he da ase compa ed o
he esul s ob ained when using posi ional encodings.
On he o he hand, we also ained he model wi h posi ional encodings always p esen
du ing aining. This was done by se ing he d opou ac o o ze o, ensu ing he model
consis en ly had access o hese ea u es. This esul ed in he gene a ion o molecules
composed o many disconnec ed agmen s. F om his esul we can conclude wha , when
he model was condi ioned on posi ional encodings du ing aining, i elied hea ily on
his in o ma ion o unde s and he g aph’s s uc u e. Because o ha , when posi ional
encodings we e absen du ing sampling, he model s uggled o gene a e well-connec ed
g aphs. This esul emphasizes he impo ance o posi ional encodings in cap u ing he
molecula s uc u e connec i i y.
Simila ly, we ha e also measu ed he effec o he weigh o he loss o he edges.
When aining wi h a loss ha assigns equal weigh o bo h nodes and edges, he numbe
o alid molecules gene a ed d opped o 46.26%. This sugges s ha he model gene a es
ewe alid molecules, likely due o inco ec connec ions be ween nodes. This empi ically
demons a es ha edges a e he mos challenging aspec o he model o gene a e wi hou
comp omising molecula alidi y. By inc easing he weigh o he edges loss o 10, we
achie ed he bes esul s wi h he highes alidi y sco e. This ou come aligns wi h he
obse a ion ha gene a ing co ec bonds is he mos difficul ask, as in alid molecules
o en con ain inco ec bonds. This issue is pa icula ly e iden in he gene a ion o
molecula ings, whe e he model s uggles due o he high numbe o edges in ol ed.
We also in es iga ed he effec o he limi dis ibu ion on he model’s pe o mance.
Specifically, we ained he model using a uni o m dis ibu ion, whe e each node and edge
was sampled wi h he same p obabili y, ins ead o using a limi dis ibu ion de i ed om
he da ase . Fo his expe imen , we used he ZINC250k da ase . Al hough he bond and
edge dis ibu ions o he gene a ed molecules we e again close o hose o he da ase , he
esul ing molecules we e less accu a e compa ed o hose gene a ed using he ma ginal
dis ibu ions as he limi dis ibu ion. In pa icula , he pe cen age o alid molecules
30
d opped o 42.84%, and he FCD sco e inc eased o 11.84, indica ing a no able decline in
simila i y o he da ase . As shown in Table 4 and Figu es 16 and 17 in he appendix, he
dis ibu ions ob ained unde he uni o m limi dis ibu ion ail o ep oduce he da ase
me ics as effec i ely as hose ob ained using he ma ginal dis ibu ions (Figu es 9 and 10).
These findings ein o ce ha adop ing he da ase dis ibu ion as he limi dis ibu ion
significan ly enhances he gene a ion p ocess, leading o imp o ed alidi y and close
esemblance o he o iginal da ase .
We also a emp ed o ain he model using he KL di e gence e m (equa ion 3.16)
om he lowe bound as pa o he loss unc ion. Howe e , we obse ed ha his e m,
being e y small, led o NaN alues du ing aining, making his aining impossible. As
a esul , we op ed o ain he model solely wi h he c oss-en opy loss, which success ully
s abilized he aining p ocess.
The esul s o all he abla ion expe imen s can be seen in Table 4. I is impo an
o no e ha he me ics we e no calcula ed o he model ained adding always he
posi ional encodings. This is because, in his case, he alid molecules gene a ed we e
limi ed o e y small agmen s, o en consis ing o jus one o wo a oms. As a esul ,
he me ics in his scena io a e no meaning ul.
Me ic Wi hou PE Equal weigh Uni o m
Validi y 46.73% 46.26% 42.84%
Uniqueness 99.97% 99.98% 99.95%
No el y 100.00% 100.00% 99.99%
FCD 12.20 11.98 11.84
Table 4: E alua ion me ics o he abla ion expe imen s. The fi s expe imen in ol es
aining he model wi hou any posi ional encoding, he second expe imen uses a loss
unc ion wi h equal weigh s o bo h nodes and edges, and he final expe imen ains he
model using a uni o m dis ibu ion as he limi dis ibu ion.
31
Chap e 5
Analysis o he sus ainabili y and
e hical implica ions
The de elopmen o he gene a i e model has in ol ed significan en i onmen al, eco-
nomic, and social conside a ions. F om an en i onmen al s andpoin , he aining p ocess
equi ed mul iple expe imen s wi h a ying model a chi ec u es and efinemen s o ensu e
op imal pe o mance. This esul ed in ex ensi e ene gy consump ion o e app oxima ely
1821 hou s o aining, leading o an es ima ed emission o 275.34 kg o CO2
1. The
economic impac is also conside able, as he aining p ocess elied on high-pe o mance
A30 wi h an Ampe e a chi ec u e, which come wi h subs an ial cos s bo h in e ms o
ha dwa e acquisi ion and elec ici y consump ion.
The execu ion phase o he p ojec p esen s a diffe en pe spec i e. En i onmen ally,
he model’s abili y o compu a ionally gene a e new d ugs has he po en ial o educe he
need o labo a o y-based es ing, which o en in ol es significan plas ic consump ion and
o he non- ecyclable ma e ials. The model’s efficiency in explo ing a as chemical space
compu a ionally minimizes physical was e and expe imen al e o s. Economically, he
p ecise benefi s a e ied o educed ma e ial cos s and as e d ug candida e gene a ion.
Socially, he use o compu a ional ools educes he need o expe imen al molecule e i-
fica ion, a p ocess ha would be highly inefficien and labo -in ensi e gi en he as ness
o chemical space.
Rega ding he isks and limi a ions o he p ojec , he economic limi a ions a e clea ,
as he model could no be ained on la ge da ase s due o ime cons ain s and he
limi ed a ailabili y o compu a ional esou ces. Socially, while he model is designed o
d ug disco e y in he pha maceu ical indus y, he e is a isk ha such echnology could be
misused by malicious ac o s o gene a e ha m ul subs ances, aising e hical conce ns abou
dual-use esea ch [51]. Fu he mo e, he eliance on ad anced compu a ional esou ces
aises ques ions abou he accessibili y and equi able dis ibu ion o he p ojec , since i
could only be effec i ely used wi h sufficien echnological in as uc u e.
In alignmen wi h p o essional e hical s anda ds, he wo k conduc ed in his p ojec
espec s he p inciples o scien ific in eg i y, anspa ency, and esponsible use o ech-
nology. The me hodologies and da ase s ha e been chosen and documen ed wi h ca e o
a oid po en ial misuse, and he de elopmen p ocess ollows guidelines o sa e and e hical
AI use, emphasizing posi i e socie al impac while mi iga ing isks.
1h ps://calcula o .linkedda a.es/
32
Chap e 6
Conclusions
To conclude, ou expe imen s demons a e he effec i eness o he p oposed model in gen-
e a ing alid and di e se molecules ac oss mul iple da ase s, including QM9, ZINC250k,
and MOSES. The model’s pe o mance was assessed using a ious e alua ion me ics, in-
cluding alidi y, uniqueness, no el y, and FCD, and compa isons wi h o he s a e-o - he-
a gene a i e models. The esul s indica e ha ou model is able o gene a e molecules
ha closely esemble he o iginal da ase ’s dis ibu ion, pa icula ly in e ms o a om
and bond dis ibu ions, as well as o he molecula p ope ies dis ibu ions, while also
main aining high no el y and uniqueness sco es.
The model success ully lea ned he dis ibu ion o he QM9 da ase , and i s pe o -
mance imp o ed when es ed on la ge da ase s, such as ZINC250k and MOSES, which
p o ided mo e a ied and complex molecula s uc u es o aining. These esul s sug-
ges ha he models abili y o gene alize and gene a e no el molecules imp o es as he
da ase size inc eases.
Fo u u e wo k, we plan o enhance he model by aining i on e en la ge and mo e
di e se da ase s. This will allow us o assess whe he he model can cap u e mo e in ica e
molecula ea u es and po en ially imp o e i s pe o mance in e ms o bo h alidi y and
di e si y. Specifically, we aim o ain he model on a highe ZINC subse , con aining 9.26
million molecules. This will enable us o measu e he imp o emen o he model when
inc easing he size o he da ase .
We also in end o explo e downs eam applica ions o he model o gene a ing molecules
ailo ed o specific pu poses. A key ocus will be on p ope y-condi ioned molecule gen-
e a ion, enabling scaffold hopping o c ea e no el compounds wi h simila p ope ies o
exis ing molecules while a oiding pa en es ic ions.
Ano he impo an di ec ion is adap ing he model o gene a e molecules wi h 3D
a omic posi ions. This ex ension will allow he model o p edic no only he a omic and
bond s uc u es bu also hei spa ial a angemen s. Such ad ancemen s a e pa icula ly
ele an o gene a ing small molecules capable o binding effec i ely o a ge p o eins,
pa ing he way o designing compounds wi h he apeu ic po en ial o modula e p o ein
ac i i y.
33
Bibliog aphy
[1] F. Eijkelboom, G. Ba osh, C. A. Naesse h, M. Welling, and J. W. an de Meen .
Va ia ional flow ma ching o g aph gene a ion. a Xi p ep in a Xi :2406.04843,
2024.
[2] H. Tang, C. Li, S. Kamei, Y. Yamanishi, and Y. Mo imo o. Molecula gen-
e a i e ad e sa ial ne wo k wi h mul i-p ope y op imiza ion. a Xi p ep in
a Xi :2404.00081, 2024.
[3] C. Vignac, I. K awczuk, A. Si audin, B. Wang, V. Ce he , and P. F ossa d. DiG ess:
Disc e e denoising diffusion o g aph gene a ion. In The Ele en h In e na ional
Con e ence on Lea ning Rep esen a ions, 2023.
[4] D. Polyko skiy, A. Zheb ak, B. Sanchez-Lengeling, S. Golo ano , O. Ta ano ,
S. Belyae , R. Ku bano , A. A amono , V. Aladinskiy, M. Veselo , A. Kadu in,
S. Johansson, H. Chen, S. Nikolenko, A. Aspu u-Guzik, and A. Zha o onko . Molec-
ula Se s (MOSES): A Benchma king Pla o m o Molecula Gene a ion Models.
F on ie s in Pha macology, 2020.
[5] J. P. Hughes, S. Rees, S. B. Kalindjian, and K. L. Philpo . P inciples o ea ly d ug
disco e y. B i ish Jou nal o Pha macology, 162:1239–1249, 2011.
[6] K. Bibe , A. Bha acha ya, B. M. Campbell, J. R. Pi o, M. Rohe, R. G. W. S aal,
R. V. Talanian, and T. Mölle . Mic oglial d ug a ge s in AD: Oppo uni ies and
challenges in d ug disco e y and de elopmen . F on ie s in Pha macology, 10:840,
2019.
[7] M. S. Cohen, T. Hol zman, J. Michel, and J. Ma shall. Small molecules as ools o
modula ing p o ein ac i i y. Na u e Re iews D ug Disco e y, 3:927–938, 2004.
[8] H. O. Rasul, D. D. Gha ou , B. K. Aziz, B. A. Hassan, T. A. Rashid, and A. Ki ak.
Decoding d ug disco e y: Explo ing A- o-Z in silico me hods o beginne s. Applied
Biochemis y and Bio echnology, pages 1–51, 2024.
[9] S. Chak abo y, P. Kayas ha, and R. Ramak ishnan. The chemical space o b, n-
subs i u ed polycyclic a oma ic hyd oca bons: Combina o ial enume a ion and high-
h oughpu fi s -p inciples modeling. The Jou nal o Chemical Physics, 150(11),
2019.
[10] R. O. Fox, P. A. E ans, and C. M. Dobson. Mul iple con o ma ions o a p o ein
demons a ed by magne iza ion ans e NMR spec oscopy. Na u e, 320(6058):192–
194, 1986.
34
[11] A. V. Sadybeko and V. Ka i ch. Compu a ional app oaches s eamlining d ug
disco e y. Na u e, 616(7958):673–685, 2023.
[12] Y. Shimizu, M. Oh a, S. Ishida, e al. AI-d i en molecula gene a ion o no -pa en ed
pha maceu ical compounds using wo ld open pa en da a. Jou nal o Chemin o ma -
ics, 15(1):120, 2023.
[13] P. J. Guns and J. Joossens. In ellec ual p ope y managemen in academic d ug
disco e y: Wha a e he challenges? Pha maceu ical Pa en Analys , 5(2):83–85,
2016.
[14] Y. Hu, D. S ump e, and J. Bajo a h. Recen ad ances in scaffold hopping: minipe -
spec i e. Jou nal o Medicinal Chemis y, 60(4):1238–1246, 2017.
[15] K. K. Mak, Y. H. Wong, and M. R. Pichika. A ificial in elligence in d ug disco e y
and de elopmen . In D ug Disco e y and E alua ion: Sa e y and Pha macokine ic
Assays, pages 1–14. Sp inge , 2023.
[16] D. Rigoni, N. Na a in, and A. Spe du i. Condi ional cons ained g aph a ia ional
au oencode s o molecule design. In 2020 IEEE Symposium Se ies on Compu a ional
In elligence (SSCI), pages 729–736. IEEE, 2020.
[17] C. Bilodeau, W. Jin, T. Jaakkola, R. Ba zilay, and K. F. Jensen. Gene a i e models
o molecula disco e y: Recen ad ances and challenges. WIREs Compu a ional
Molecula Science, 2022.
[18] D. C. El on, Z. Boukou alas, M. D. Fuge, and P. W. Chung. Deep lea ning o molec-
ula designa e iew o he s a e o he a . Molecula Sys ems Design & Enginee ing,
4(4):828–849, 2019.
[19] D. Weininge . SMILES, a chemical language and in o ma ion sys em. 1. in oduc ion
o me hodology and encoding ules. Jou nal o Chemical In o ma ion and Compu e
Sciences, 28(1):31–36, 1988.
[20] W. Jin, R. Ba zilay, and T. Jaakkola. Junc ion ee a ia ional au oencode o
molecula g aph gene a ion. In In e na ional Con e ence on Machine Lea ning, pages
2323–2332. PMLR, 2018.
[21] M. J. Kusne , B. Paige, and J. M. He nández-Loba o. G amma a ia ional au oen-
code . In In e na ional Con e ence on Machine Lea ning, pages 1945–1954. PMLR,
2017.
[22] N. De Cao and T. Kip . MolGAN: An implici gene a i e model o small molecula
g aphs. ICML 2018 wo kshop on Theo e ical Founda ions and Applica ions o Deep
Gene a i e Models, 2018.
[23] J. N. Wu, T. Wang, Y. Chen, L. J. Tang, H. L. Wu, and R. Q. Yu. -SMILES: a
agmen -based molecula ep esen a ion amewo k o de no o ligand design. Na u e
Communica ions, 15(1):4993, 2024.
[24] M. K enn, F. Häse, A. Nigam, P. F iede ich, and A. Aspu u-Guzik. Sel - e e encing
embedded s ings (SELFIES): A 100% obus molecula s ing ep esen a ion. Ma-
chine Lea ning: Science and Technology, 1(4):045024, 2020.
35
[25] H. Huang, L. Sun, B. Du, and W. L . Condi ional diffusion based on disc e e g aph
s uc u es o molecula g aph gene a ion. In P oceedings o he AAAI Con e ence
on A ificial In elligence, olume 37, pages 4302–4311, 2023.
[26] H. Chen, C. Xu, L. Zheng, Q. Zhang, and X. Lin. Diffusion-Based G aph Gene a-
i e Me hods. IEEE T ansac ions on Knowledge & Da a Enginee ing, 36:7954–7972,
2024.
[27] Y. Chen, J. Liu, L. Peng, Y. Wu, Y. Xu, and Z. Zhang. Au o-encoding a ia ional
bayes. Camb idge Explo a ions in A s and Sciences, 2(1), 2024.
[28] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion p obabilis ic models. Ad ances in
Neu al In o ma ion P ocessing Sys ems, 33:6840–6851, 2020.
[29] J. Song, C. Meng, and S. E mon. Denoising diffusion implici models. a Xi p ep in
a Xi :2010.02502, 2020.
[30] J. Aus in, D. D. Johnson, J. Ho, D. Ta low, and R. Van Den Be g. S uc u ed
denoising diffusion models in disc e e s a e-spaces. Ad ances in Neu al In o ma ion
P ocessing Sys ems, 34:17981–17993, 2021.
[31] A. Q. Nichol and P. Dha iwal. Imp o ed denoising diffusion p obabilis ic models. In
In e na ional Con e ence on Machine Lea ning, pages 8162–8171. PMLR, 2021.
[32] A. Vaswani, N. Shazee , N. Pa ma , J. Uszko ei , L. Jones, A. N. Gomez, . Kaise , and
I. Polosukhin. A en ion is all you need. Ad ances in Neu al In o ma ion P ocessing
Sys ems, 2017.
[33] A. Bansal, E. Bo gnia, H-M Chu, J. Li, H. Kazemi, F. Huang, M. Goldblum, J. Geip-
ing, and T. Golds ein. Cold diffusion: In e ing a bi a y image ans o ms wi hou
noise. Ad ances in Neu al In o ma ion P ocessing Sys ems, 36, 2024.
[34] J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun.
G aph neu al ne wo ks: A e iew o me hods and applica ions. AI Open, 1:57–81,
2020.
[35] J. Gilme , S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neu al message
passing o quan um chemis y. In In e na ional Con e ence on Machine Lea ning,
pages 1263–1272. PMLR, 2017.
[36] M. Dehghani, J. Djolonga, B. Mus a a, P. Padlewski, J. Heek, J. Gilme , A. P.
S eine , M. Ca on, R. Gei hos, I. Alabdulmohsin, e al. Scaling ision ans o me s
o 22 billion pa ame e s. In In e na ional Con e ence on Machine Lea ning, pages
7480–7512. PMLR, 2023.
[37] V. P. Dwi edi and X. B esson. A gene aliza ion o ans o me ne wo ks o g aphs.
AAAI Wo kshop on Deep Lea ning on G aphs: Me hods and Applica ions, 2021.
[38] V. P. Dwi edi, A. T. Luu, T. Lau en , Y. Bengio, and X. B esson. G aph neu al
ne wo ks wi h lea nable s uc u al and posi ional ep esen a ions. In In e na ional
Con e ence on Lea ning Rep esen a ions, 2022.
36
[39] K. P eue , P. Renz, T. Un e hine , S. Hoch ei e , and G. Klambaue . F éche
ChemNe dis ance: a me ic o gene a i e models o molecules in d ug disco e y.
Jou nal o Chemical In o ma ion and Modeling, 58(9):1736–1741, 2018.
[40] J. J. I win, T. S e ling, M. M. Mysinge , E. S. Bols ad, and R. G. Coleman. ZINC:
A ee ool o disco e chemis y o biology. Jou nal o Chemical In o ma ion and
Modeling, 52(7):1757–1768, 2012.
[41] Y. Wang, S. H. B yan , T. Cheng, J. Wang, A. Ginduly e, B. A. Shoemake , P. A.
Thiessen, S. He, and J. Zhang. PubChem BioAssay: 2017 upda e. Nucleic Acids
Resea ch, 45(D1):D955–D963, 2016.
[42] A. P. Ben o, A. Gaul on, A. He sey, L. J. Bellis, J. Chambe s, M. Da ies, F. A.
K üge , Y. Ligh , L. Mak, S. McGlinchey, M. Nowo ka, G. Papada os, R. San os,
and J. P. O e ing on. The ChEMBL bioac i i y da abase: an upda e. Nucleic Acids
Resea ch, 42(D1):D1083–D1090, 2013.
[43] M. Heusel, H. Ramsaue , T. Un e hine , B. Nessle , and S. Hoch ei e . Gans ained
by a wo ime-scale upda e ule con e ge o a local nash equilib ium. Ad ances in
Neu al In o ma ion P ocessing Sys ems, 30, 2017.
[44] G. R. Bicke on, G. V. Paolini, J. Besna d, So el Mu esan, and A. L. Hopkins.
Quan i ying he chemical beau y o d ugs. Na u e Chemis y, 4(2):90–98, 2012.
[45] S. A. Wildman and G. M. C ippen. P edic ion o physicochemical pa ame e s by
a omic con ibu ions. Jou nal o Chemical In o ma ion and Compu e Sciences,
39(5):868–873, 1999.
[46] P. E l and A. Schuffenhaue . Es ima ion o syn he ic accessibili y sco e o d ug-like
molecules based on molecula complexi y and agmen con ibu ions. Jou nal o
Chemin o ma ics, 1(1):8, 2009.
[47] R. Ramak ishnan, P. O. D al, M. Rupp, and O. A. on Lilien eld. Quan um chemis y
s uc u es and p ope ies o 134 kilo molecules. Scien ific Da a, 1, 2014.
[48] C. Niu, Y. Song, J. Song, S. Zhao, A. G o e , and S. E mon. Pe mu a ion in a ian
g aph gene a ion ia sco e-based gene a i e modeling. In In e na ional Con e ence
on A ificial In elligence and S a is ics, pages 4474–4484. PMLR, 2020.
[49] D. Liu, S. Chen, S. Zheng, S. Zhang, and Y. Yang. SE(3) equi alen g aph a en ion
ne wo k as an ene gy-based model o p o ein side chain con o ma ion. In 2023
IEEE In e na ional Con e ence on Bioin o ma ics and Biomedicine (BIBM), pages
120–123. IEEE, 2023.
[50] X. Wang, T. Hu, X. Ren, J. Sun, K. Liu, and M. Zhang. T ansVae: A no el a i-
a ional sequence- o-sequence amewo k o semi-supe ised lea ning and di e si y
imp o emen . In 2021 In e na ional Join Con e ence on Neu al Ne wo ks (IJCNN),
pages 1–8, 2021.
[51] W. Fel us and J. Smi h. E hical and secu i y implica ions o dual-use echnologies
in d ug disco e y. Jou nal o Biosecu i y and E hics, 5(2):123–134, 2017.
37
A Da ase s
The QM9 da ase [47] is a widely used molecula da ase o aining gene a i e models
and e alua ing p ope y p edic ion asks, wi h 134000 s able small o ganic molecules.
These molecules a e composed jus o Ca bon (C), Hyd ogen (H), Oxygen (O), Ni ogen
(N), and Fluo ine (F). The QM9 da ase ocuses on molecules wi h up o jus nine hea y
a oms, ep esen ing a ela i ely simple chemical space wi h limi ed s uc u al complexi y.
On he o he hand, bo h MOSES [4] and ZINC250k [40] a e mo e complex da ase s,
de i ed om he ZINC da abase, a ee public esou ce o ligand disco e y con aining
o e 230 million comme cially a ailable molecule. ZINC250k con ains 250000 a ailable
molecules wi h up o 38 hea y a oms, including Hyd ogen (H), Ca bon (C), Sul u (S),
Oxygen (O), Ni ogen (N), Chlo ine (Cl), Fluo ine (F), B omine (B ), Iodine (I) and
Phospho us (P). MOSES is a bigge subse , con aining 1.94 million molecules, fil e ed
using diffe en c i e ia. These molecules con ain up o 27 hea y a oms, including Hyd ogen
(H), Ca bon (C), Sul u (S), Oxygen (O), Ni ogen (N), Chlo ine (Cl), Fluo ine (F) and
B omine (B ). This da ase is also fil e ed so ha i does no con ain cycles la ge wi h
mo e han 8 a oms.
Due o he na u e o he da ase s, me ics such as QED, logP, and o he s a e only
measu ed o he gene a ed samples om he MOSES and ZINC250k da ase s, as bo h
con ain d ug-like molecules. As a esul , measu ing hese me ics is meaning ul o assess-
ing he quali y and ele ance o he gene a ed molecules in hese da ase s. Howe e , since
QM9 consis s o simple molecules ha a e no d ug-like, hese me ics a e no ele an
o use ul o e alua ing he model’s pe o mance on his da ase .
38
(a) SA dis ibu ions.
(b) Molecula weigh s dis ibu ions.
Figu e 17: SA and molecula weigh s dis ibu ions o he gene a ed molecules wi h a
uni o m dis ibu ion along wi h he dis ibu ions o he ZINC250k da ase .
Some o he in e es ing me ics o meassu e he pe o mance o molecule gene a ion a e
he ollowing ones:
•In e nal Di e si y (In Di ): Assesses he chemical di e si y wi hin he gene a ed
se , de ec ing issues like mode collapse whe e he model ails o explo e he ull
chemical space. A highe sco e indica es g ea e di e si y in he gene a ed molecules.
•Nea es Neighbo Simila i y (SNN): Compa es he finge p in s, a ep esen-
a ion o molecules using bina y ec o s, o gene a ed molecules wi h hei nea es
neighbo s in he aining da ase using he Jacca d index. This measu es how closely
he gene a ed molecules esemble he aining da a.
•F agmen Simila i y (F ag): E alua es how simila he dis ibu ion o molecula
agmen s in he gene a ed se is o he aining da ase . F agmen s a e ob ained
using BRICS agmen a ion, which b eaks molecules a chemically alid bonds o en
used in agmen -based d ug disco e y.
•Scaffold Simila i y (Sca ): Compa es scaffolds, he co e s uc u al amewo ks o
molecules, be ween he gene a ed se and he aining da ase . Scaffolds ep esen
he molecula backbone a e emo ing unc ional g oups and deco a ions.
The agmen and scaffold simila i y me ics bo h assess molecula simila i y by b eak-
ing molecules in o smalle componen s, bu hey diffe in hei decomposi ion me hods.
F agmen simila i y uses BRICS agmen a ion based on chemically alid bond-b eaking
ules. This is commonly used in agmen -based d ug disco e y and he ha a e b oken
45
a e o en hose ha can be easily e- o med in chemical syn hesis. On he o he hand,
he scaffold simila i y uses he scaffolds o he molecules, which a e usually e e ed as he
co e s uc u al amewo ks o molecules. They a e ob ained a e emo ing side chains
and unc ional g oups.
We also calcula e he Wasse s ein-1 dis ance (Ea h Mo e s Dis ance) be ween he
molecula p ope ies o he gene a ed se and he aining da ase . All hese me ics a e
only compu ed o MOSES da ase because he e e ence alues o hese me ics in some
models a e de i ed om he MOSES pape , and he models hemsel es a e ained on
MOSES da ase s.
The esul s ob ained can be seen in Table 5. I can be seen ha he esul s ob ained
wi h ou model a e simila o he ones ob ained wi h one o he s a e-o - he-a models.
Howe e , as explained be o e, ou model is able o ob ain a highe numbe o no el
molecules han his model.
Me ic Ca Mol VAE
SNN 0.47 0.63
F agmen simila i y (F ag) 1.00 0.99
Scaffold simila i y (Scaff) 1.00 0.94
In e nal di e si y 0.53 0.86
Wasse s ein-1 dis ance QED 0.04 0.0006
Wasse s ein-1 dis ance weigh 0.09 1.8
Wasse s ein-1 dis ance SA 0.80 0.014
Wasse s ein-1 dis ance logP 0.32 0.023
Table 5: Mo e e alua ion me ics o he gene a ed samples o he MOSES da ase using
Ca Mol and VAE. The VAE esul s ha e been ob ained om [4].
C Configu a ion
Rega ding he model a chi ec u e, all he eed o wa d ne wo ks (FFN) ha appea in
i a e composed o wo linea laye s wi h a ReLU ac i a ion unc ion in be ween. The
model’s co e pa ame e s a e summa ized in Table 6.
Pa ame e Value
Numbe o laye s 10
Numbe o heads 8
Hidden dim 128
Ou dim 128
D opou 0.1
Table 6: Model pa ame e s used o ain he model o he diffe en da ase s.
The model con ains a o al o 370189 ainable pa ame e s.
In e ms o he diffusion p ocess, we conside a o al o T=300 imes eps. The andom
walk posi ional encodings a e added as lea nable ea u es wi h a d opou ac o o 0.7.
The op imize used o aining is AdamW, a a ian o he Adam op imize wi h
weigh decay. The lea ning a e has been se o 1e-3 wi h a weigh decay o 0.001 o
46
QM9 and ZINC250k da ase while i has been lowe ed o 1e-4 o MOSES da ase .
D Molecula gene a ion
When he gene a ion p ocess ends, wo ma ices a e ob ained: a nodes ma ix o shape
(bs, n, dx), whe e bs is he numbe o samples gene a ed, n is he numbe o a oms, and
dx is he numbe o dis inc a om ypes and an edges ma ix, and an edge ma ix o shape
(bs, n, n, de), whe e de ep esen s he numbe o possible bond ypes. These ma ices
ep esen he p obabili ies o assigning a specific a om ype o each node and a bond ype
o each edge.
To gene a e a alid g aph, we fi s sample he a om ypes and bond ypes by se-
lec ing he mos p obable ones. Once he a om ypes and bond ypes a e sampled, we
selec he la ges connec ed componen om he g aph, ensu ing ha only he p ima y
connec ed agmen is kep . This is impo an because, in molecula s uc u es, mul iple
disconnec ed agmen s would no ep esen a alid single molecule.
A e wa ds, he molecule unde goes a sani iza ion p ocess using RDKi , a widely used
open-sou ce oolki o chemin o ma ics. This p ocess in ol es a se ies o checks and co -
ec ions designed o ensu e ha he gene a ed molecule is chemically alid and con o ms
o s anda d chemical p inciples. This sani iza ion p ocess is he one ha de e mines
whe he he gene a ed molecule is alid o no , by e i ying i s s uc u e and making
necessa y adjus men s.
47
E Examples o gene a ed molecules
Figu e 18: Examples o gene a ed alid molecules aining he model wi h he QM9
da ase .
Figu e 19: Examples o gene a ed alid molecules aining he model wi h he ZINC250k
da ase .
Figu e 20: Examples o gene a ed alid molecules aining he model wi h he MOSES
da ase .
48
F Equi a iance
A unc ion is conside ed equi a ian wi h espec o a ans o ma ion g oup Gi , o an
inpu xand ans o ma ion g∈G, he ollowing condi ion holds
(g·x) = g· (x).(1)
In he con ex o machine lea ning, equi a iance e e s o he p ope y o a model
whe e i s ou pu changes p edic ably unde specific ans o ma ions applied o he inpu .
Specifically, when a ans o ma ion is applied o he inpu da a, he model’s ou pu should
ans o m in a co esponding manne .
Ou gene a i e model is node equi a ian since bo h i s backbone a chi ec u e and
loss unc ion a e designed o p ese e his ela ionship.
49
