Transformers in Time Series Forecasting: A Systematic Literature Review
Abstract
The growing interest in transformers for time series forecasting has triggered an exponential surge in publications, making it increasingly challenging to organize and understand emerging trends and discern future directions. This thesis is a systematic literature review of the subject, that aims to study how the different components and aspects of the transformers evolved, finding trends, and highlighting gaps and future directions. The review focuses on papers in English, peer-reviewed, published in reputable sources, and that focus exclusively on transformer models for time series forecasting. It spans sources such as ACM, ARXiv, Elsevier, MDPI, OpenReview, AAAI, IJCAI, IEEEExplore, NeurIPS, Springer, epubs, and Jstage, from December of 2023 to April of 2024 – selecting a total of 99 papers. To assess the risk of bias, statements regarding conflicts of interest and sources of funding were examined. Publication bias was evaluated by comparing transformer results over time. The reviewed papers reveal a preference for nine datasets in specific but a lot of papers still use unique private datasets, with some variation of the time-horizons used as well as the metrics. Recent advancements showcase innovations in Input Representation (Positional Encoding, Segmentation, Decomposition, and Covariates), Modelling (Attention Mechanism, Feature Selection, Channel Dependence or Independence, Multi-scale and Hierarchical approaches, and Hybridization), Model Optimization (Regularization, Loss function, and Normalization), and Types of Learning. These advancements aim to enhance model adaptability, accuracy, robustness, and computational efficiency, as well as to improve the capture of local context and temporal relationships at different granularities. However, there is a limited understanding of transformer performance across diverse domains and datasets, as most studies focus on tailored models rather than cross-domain evaluations. The lack of standardized benchmarks complicates model assessment, necessitating more transparent and explainable models. Scalability and computational efficiency remain major concerns, especially for large-scale and real-time implementations.
Full text
Mas e Deg ee P og am in
Da a Science and Ad anced Analy ics
T ans o me s in Time Se ies Fo ecas ing
A Sys ema ic Li e a u e Re iew
Diogo Miguel Goula Fe ei a Pe ei a da Sil a
Mas e Thesis
p esen ed as pa ial equi emen o ob aining a Mas e ’s Deg ee in Da a Science and Ad anced Analy ics
NOVA In o ma ion Managemen School
Ins i u o Supe io de Es a ís ica e Ges ão de In o mação
Uni e sidade No a de Lisboa
MDSAA
NOVA In o ma ion Managemen School
Ins i u o Supe io de Es a ís ica e Ges ão de In o mação
Uni e sidade No a de Lisboa
T ans o me s in Time Se ies Fo ecas ing
A Sys ema ic Li e a u e Re iew
by
Diogo Miguel Goula Fe ei a Pe ei a da Sil a
Mas e Thesis p esen ed as pa ial equi emen o ob aining he Mas e ’s deg ee in Da a
Science and Ad anced Analy ics, wi h a specializa ion in Da a Science.
Supe ised by
Robe o Hen iques, PhD, No a IMS
July, 2024
i
STATEMENT OF INTEGRITY
I he eby decla e ha ing conduc ed his academic wo k wi h in eg i y. I con i m ha I ha e
no used plagia ism o any o m o undue use o in o ma ion o alsi ica ion o esul s along
he p ocess leading o i s elabo a ion. I u he decla e ha I ha e ully acknowledged he
Rules o Conduc and Code o Hono om he NOVA In o ma ion Managemen School.
[Lisbon, 11/07/2024]
ii
DEDICATION
Pa a o meu a ô An ónio, que b ilha semp e, mesmo na ad e sidade. Ten a ei semp e segui
os eus passos.
iii
ACKNOWLEDGEMENTS
Ag adeço à minha amília, o meu p imei o pila , o meu ab igo e a minha casa,
independen emen e de udo.
Ag adeço à minha mãe, que semp e ez os impossí eis po mim, e po a minha educação e
sido semp e uma das suas g andes p io idades. Ob igado po odas as po as que me
ab is e, a eu cus o. Onde que que chegue, se á g aças a me e es e guido nos eus omb os.
Ag adeço aos meus a ós que o am incansá eis em nunca me al a nada, que em g ande
plano que po de ás das cenas, em ges os pequenos e g andes, semp e consis en es e
p esen es. E que, apesa de não pe cebe em nada des e assun o, o am incansá eis a
acompanha a esc i a des a ese, semp e p eocupados e a que e sabe como es a a a
co e , e a é a en a am le .
Ag adeço à Ana po , como boa i mã, se semp e di e a e hones a e me aze de ol a à
ealidade quando p eciso.
Ag adeço ao Sé gio, que é semp e on e de calma e segu ança, e a anja semp e empo e
boa disposição quando necessá io, semp e dispos o a ajuda .
Ag adeço aos meus amigos que me man i e am são nes e ano di ícil, ocês sabem quem
são.
Ag adeço ao meu o ien ado , P o esso Robe o Hen iques, po e ido semp e
disponibilidade quando oi necessá io, p incipalmen e na al u a desa ian e em que i e que
muda de ema de ese.
E ag adeço à No a IMS po me e dado a opo unidade de, nes e mes ado, descob i a
minha e dadei a ocação.
i
ABSTRACT
The g owing in e es in ans o me s o ime se ies o ecas ing has igge ed an exponen ial
su ge in publica ions, making i inc easingly challenging o o ganize and unde s and
eme ging ends and disce n u u e di ec ions. This hesis is a sys ema ic li e a u e e iew o
he subjec , ha aims o s udy how he di e en componen s and aspec s o he
ans o me s e ol ed, inding ends, and highligh ing gaps and u u e di ec ions. The e iew
ocuses on pape s in English, pee - e iewed, published in epu able sou ces, and ha ocus
exclusi ely on ans o me models o ime se ies o ecas ing. I spans sou ces such as ACM,
ARXi , Else ie , MDPI, OpenRe iew, AAAI, IJCAI, IEEEExplo e, Neu IPS, Sp inge , epubs, and
Js age, om Decembe o 2023 o Ap il o 2024 – selec ing a o al o 99 pape s. To assess he
isk o bias, s a emen s ega ding con lic s o in e es and sou ces o unding we e
examined. Publica ion bias was e alua ed by compa ing ans o me esul s o e ime. The
e iewed pape s e eal a p e e ence o nine da ase s in speci ic bu a lo o pape s s ill use
unique p i a e da ase s, wi h some a ia ion o he ime-ho izons used as well as he
me ics. Recen ad ancemen s showcase inno a ions in Inpu Rep esen a ion (Posi ional
Encoding, Segmen a ion, Decomposi ion, and Co a ia es), Modelling (A en ion Mechanism,
Fea u e Selec ion, Channel Dependence o Independence, Mul i-scale and Hie a chical
app oaches, and Hyb idiza ion), Model Op imiza ion (Regula iza ion, Loss unc ion, and
No maliza ion), and Types o Lea ning. These ad ancemen s aim o enhance model
adap abili y, accu acy, obus ness, and compu a ional e iciency, as well as o imp o e he
cap u e o local con ex and empo al ela ionships a di e en g anula i ies. Howe e , he e
is a limi ed unde s anding o ans o me pe o mance ac oss di e se domains and da ase s,
as mos s udies ocus on ailo ed models a he han c oss-domain e alua ions. The lack o
s anda dized benchma ks complica es model assessmen , necessi a ing mo e anspa en
and explainable models. Scalabili y and compu a ional e iciency emain majo conce ns,
especially o la ge-scale and eal- ime implemen a ions.
KEYWORDS
T ans o me ; Time Se ies; Fo ecas ; A en ion Mechanism; Deep Lea ning
TABLE OF CONTENTS
1. In oduc ion .................................................................................................................. 1
2. Backg ound Theo y o T ans o me s ............................................................................ 3
2.1. Encode And Decode ............................................................................................ 3
2.2. Posi ional Encoding ............................................................................................... 4
2.3. Sel -A en ion ........................................................................................................ 5
2.4. Mul i-Head A en ion ............................................................................................ 6
3. Me hodology ................................................................................................................ 7
4. Pape s’ Analysis .......................................................................................................... 10
4.1. Da es, Da abases and Con e ences ..................................................................... 10
4.2. Ci a ions ............................................................................................................... 11
4.3. Code A ailabili y and Pee Re iew ...................................................................... 12
4.4. Da ase s ............................................................................................................... 13
4.5. Me ics ................................................................................................................. 15
4.6. Bias....................................................................................................................... 17
4.7. Fo ecas ing ends ............................................................................................... 19
4.8. P ep ocessing, Spli s, and Tuning ........................................................................ 20
5. E olu ion and Discussion ............................................................................................ 22
5.1. Inpu Rep esen a ion .......................................................................................... 23
5.1.1. Posi ional Embedding ................................................................................... 24
5.1.1.1. Adap i e and Lea nable Embeddings ................................................ 24
5.1.1.2. Mul i-Resolu ion Embeddings............................................................ 24
5.1.1.3. Tempo al Focused Embeddings ......................................................... 26
5.1.1.4. Adding o Con ex ual In o ma ion ..................................................... 27
5.1.1.5. Specialized Embeddings ..................................................................... 27
5.1.1.6. Discussion ........................................................................................... 28
5.1.2. Co a ia es ..................................................................................................... 29
5.1.2.1. Conca ena ion and Embedding.......................................................... 29
5.1.2.2. Specialized Embeddings ..................................................................... 29
5.1.2.3. Specialized Embeddings ..................................................................... 30
5.1.2.4. Gene ic Algo i hm In eg a ion ........................................................... 30
5.1.2.5. Discussion ........................................................................................... 30
5.1.3. Segmen s ...................................................................................................... 31
5.1.3.1. Pa ch-based Mechanisms .................................................................. 31
i
5.1.3.2. Window-based Mechanisms .............................................................. 32
5.1.3.3. Adap i e and Hyb id Segmen a ion ................................................... 33
5.1.3.4. Discussion ........................................................................................... 34
5.1.4. Decomposi ion ............................................................................................. 34
5.1.4.1. T end-Seasonali y Decomposi ion ..................................................... 34
5.1.4.2. Decomposi ion Blocks ........................................................................ 35
5.1.4.3. F equency Domain T ans o ma ions ................................................. 35
5.1.4.4. Specialized Decomposi ion ................................................................ 36
5.1.4.5. Hyb id Me hods ................................................................................. 37
5.1.4.6. P e-p ocessing and Dual-Domain S a egies ..................................... 37
5.1.4.7. Discussion ........................................................................................... 38
5.2. Modeling .............................................................................................................. 38
5.2.1. A en ion Mechanism ................................................................................... 39
5.2.1.1. Spa se Mechanisms ........................................................................... 39
5.2.1.2. Hyb id Mechanisms ........................................................................... 40
5.2.1.3. F equency Mechanisms ..................................................................... 41
5.2.1.4. Segmen -based Mechanisms ............................................................. 42
5.2.1.5. Hie a chical and Mul i-Scale Mechanisms ......................................... 43
5.2.1.6. A en ion Mechanisms o Va iable In e ac ion ................................ 43
5.2.1.7. A en ion Mechanisms wi h Specialized Techniques ........................ 44
5.2.1.8. Discussion ........................................................................................... 45
5.2.2. Fea u e Selec ion .......................................................................................... 46
5.2.2.1. Di ec .................................................................................................. 46
5.2.2.2. Indi ec ............................................................................................... 47
5.2.2.3. Discussion ........................................................................................... 47
5.2.3. Channel Dependence and Independence .................................................... 48
5.2.3.1. Channel Dependence ......................................................................... 48
5.2.3.2. Channel Independence ...................................................................... 49
5.2.3.3. Discussion ........................................................................................... 49
5.2.4. Hie a chical and Mul i-Scale App oaches .................................................... 50
5.2.4.1. Hie a chical Only ................................................................................ 50
5.2.4.2. Mul i-Scale Only ................................................................................. 51
2
Due o he g owing in e es in he a ea as well as he success o he p oposed models, he e
has been an exponen ial inc ease o published pape s, ocusing on di e en aspec s and
making i inc easingly challenging o o ganize and unde s and eme ging ends, as well as o
disce n u u e di ec ions. Addi ionally, exis ing su eys and e iews quickly became
ou da ed. This poses a signi ican challenge o hose en e ing he ield and po en ially
o e whelming e en expe s.
This hesis aims o add ess his gap by posing he esea ch ques ion: “How ha e ans o me
models e ol ed o add ess he speci ic challenges o ime se ies o ecas ing, and wha a e
he eme ging ends and u u e di ec ions in his a ea?”, wi h he esea ch objec i es being
o:
▪ Iden i y and analyse he eme ging ends, inno a ions, and me hodological
ad ancemen s in ans o me models ailo ed o ime se ies o ecas ing.
▪ Pinpoin exis ing gaps, challenges, and limi a ions in he scien i ic knowledge and o
s a e-o - he-a ans o me models
▪ P opose u u e esea ch di ec ions and po en ial echnological ad ancemen s.
The o ganiza ion o his hesis will be as ollows:
Chap e 2. Backg ound Theo y o T ans o me s: Explains he heo y behind ans o me s;
Chap e 3. Me hodology: Explaining he sea ch s a egy used o ind he pape s, he
inclusion and exclusion c i e ia, he da a collec ion p ocess, s a e he p ima y and seconda y
ou comes and he me hod o assess he isk o bias; Chap e 4. Pape s’ Analysis: Focuses on
publica ion da es, sou ces, code a ailabili y, and biases, as well as speci ic in o ma ion like
da ase s, me ics, and loss unc ions used. In e es ing bu less ele an in o ma ion is in
Appendix B (Pape s’ S udy Ex a); Chap e 5. E olu ion and Discussion: Shows he e olu ion
o ans o me s, di ided by componen o he ans o me a chi ec u e. This app oach
p o ides a clea unde s anding o how di e en componen s change ac oss s udies; Chap e
6. Conclusion and Fu u e Wo k.
Appendix A (Complexi y Equa ions): Shows he di e en complexi y equa ions shown in
he pape s and he mechanisms o each hem; Appendix B (Pape s’ S udy Ex a) – S udies
on in e es ing bu no he mos ele an cha ac e is ics – numbe o pages, numbe o paged
o he annex, numbe o e e ences used, numbe o au ho s and whe e he au ho s a e
om - we e mo ed o his appendix; Appendix C (Table o Summa y o Changes): A able
summa izing he changes made o each ans o me , whe he he code is publicly a ailable,
he mo i a ion/ issue add essed, and he e e ence; Appendix D (Agg ega ion o Resul s):
Sec ion ocused on agglome a ing he inal me ics o di e en pape s.
3
2. BACKGROUND THEORY OF TRANSFORMERS
The ans o me was p oposed by Vaswani e al. (2023), designed o Machine T ansla ion.
This sec ion ocuses on explaining he inne mechanics o he a chi ec u e, wi h he main
con ibu ions o he sec ion being he p e ious pape , L. Su e al. (2023), Tay e al. (2022),
and T. Lin e al. (2021).
Figu e 2.1 shows he a chi ec u e o he ans o me , ha uses sel -a en ion and poin -wise
ully connec ed laye s o bo h he encode and he decode .
Figu e 2.1 – A chi ec u e o he T ans o me model (Vaswani e al., 2023)
2.1. ENCODER AND DECODER
The encode is depic ed on he le -hand side o Figu e 2.1. The encode componen
comp ises wo p ima y ne wo ks: he mul i-head a en ion mechanism and he wo-laye
eed- o wa d neu al ne wo k. Each o hese sub-ne wo ks bene i s om he addi ion o
esidual connec ions, which help mi iga e he anishing g adien p oblem and enable he
ne wo k o ain deepe laye s mo e e ec i ely, as shown by He e al. (2015), Laye
no maliza ion ollows hese esidual connec ions, ensu ing ha he ou pu s a e s able and
no malized, ep esen ed as 𝐿𝑎𝑦𝑒𝑟𝑁𝑜𝑟𝑚(𝑥+𝑆𝑢𝑏𝑙𝑎𝑦𝑒𝑟(𝑥)). Addi ionally, d opou is
applied o p e en o e i ing.
4
The wo-laye eed- o wa d ne wo k (FFN) in he second pa o he encode p ocesses he
inpu ea u e ep esen a ion 𝑥 h ough he ollowing sequence o ope a ions:
𝐹𝐹𝑁(𝑥)=max(0,𝑥𝑊1+𝑏)𝑊2+ 𝑏2
Whe e 𝑊1 and 𝑊2 deno e he weigh ma ices, while 𝑏1 and 𝑏2 ep esen he bias ec o s.
The unc ion max(0,…) signi ies he u iliza ion o ReLU as he ac i a ion unc ion.
The decode is simila o he encode , albei wi h he inclusion o an ex a mul i-head
a en ion mechanism ha in e ac s wi h he encode ou pu . The decode consis s o h ee
p ima y sub-ne wo ks: wo simila o he encode ’s sub-ne wo ks and a hi d ha pe o ms
encode -decode a en ion.
In he encode -decode a en ion componen , he inpu que y 𝑄 is de i ed om he ou pu
o he p eceding mul i-head a en ion mechanism, while he keys 𝐾 and alues 𝑉 a e linea ly
ans o med ou pu s (Memo y) om he encode . This s uc u e ensu es ha he decode
can e ec i ely le e age he encoded inpu in o ma ion o p oduce accu a e and con ex ually
ele an ou pu s.
2.2. POSITIONAL ENCODING
In he ealm o modelling ex - ela ed da a, he ini ial s ep in ol es ec o izing he ex .
T adi ional me hods such as one-ho encoding, bag-o -wo ds models, and TF-IDF a e
equen ly used o ex ep esen a ion. Howe e , in deep lea ning, i is mo e common o
associa e indi idual wo ds o okens wi h a low-dimensional, dense ec o space using an
embedding laye . This p ocess is e e ed o as oken embedding o wo d embedding.
In a chi ec u es like Con olu ional Neu al Ne wo ks (CNNs) o Recu en Neu al Ne wo ks
(RNNs), he ex ec o iza ion phase is o en su icien as hese ne wo ks inhe en ly cap u e
empo al ea u es. CNNs cap u e local ea u es while RNNs cap u e empo al sequences.
Howe e , his does no apply o T ans o me ne wo ks, which lack ecu sion and
con olu ion. T ans o me s ely solely on a sel -a en ion mechanism, which ope a es
h ough linea ans o ma ions by mul iplying mul iple ma ices. This means ha wi hou
addi ional mechanisms, T ans o me s a e agnos ic o he o de o he inpu sequence,
leading o po en ial loss o he o iginal ex sequence s uc u e.
To add ess he lack o inhe en sequence o de in T ans o me s, posi ional encoding is
added o he inpu embeddings. Posi ional encoding p o ides he model wi h in o ma ion
abou he posi ion o each oken in he sequence, hus p ese ing he sequence o de . The
o iginal T ans o me employs sinusoidal posi ional encodings. These encodings a e designed
o injec in o ma ion abou he ela i e and absolu e posi ions o okens in he sequence.
The encoding ec o s a e o he same dimensionali y (𝑑𝑚𝑜𝑑𝑒𝑙) as he inpu embeddings,
enabling elemen -wise addi ion. The sinusoidal posi ional encodings a e de ined as:
5
𝑃𝑜𝑠𝐸𝑚𝑏(𝑝𝑜𝑠,2𝑖)= sin( 𝑝𝑜𝑠
100002𝑖/𝑑𝑚𝑜𝑑𝑒𝑙); 𝑃𝑜𝑠𝐸𝑚𝑏(𝑝𝑜𝑠,2𝑖+1)= 𝑐𝑜𝑠( 𝑝𝑜𝑠
100002𝑖/𝑑𝑚𝑜𝑑𝑒𝑙)
He e, 𝑝𝑜𝑠 ep esen s he posi ion, and 𝑖 ep esen s he dimension. Each dimension o he
posi ional encoding co esponds o a sinusoid, wi h wa eleng hs o ming a geome ic
p og ession om 2𝜋 o 1000∙2𝜋. The 2 equa ions ep esen he posi ional encoding o
posi ion 𝑝𝑜𝑠 a he 2𝑖 - h dimension (e en index) and he (2𝑖+1)- h dimension (odd index).
This design allows he model o a end o okens by ela i e posi ions, as any ixed o se 𝐾
can be ep esen ed as a linea unc ion o he posi ional encoding.
2.3. SELF-ATTENTION
The sel -a en ion mechanism is a undamen al componen o he T ans o me a chi ec u e,
enabling he model o cap u e dependencies be ween di e en posi ions in a sequence.
Unlike ecu en and con olu ional neu al ne wo ks, sel -a en ion allows o di ec
connec ions be ween all okens in a sequence, p o iding a powe ul me hod o model long-
ange dependencies e icien ly.
An a en ion unc ion can be desc ibed as mapping a que y and a se o key- alue pai s o an
ou pu , whe e he que y, keys, alues, and ou pu a e all ec o s. The ou pu is compu ed as
a weigh ed sum o he alues, whe e he weigh assigned o each alue is de e mined by a
compa ibili y unc ion o he que y wi h he co esponding key.
The speci ic a en ion mechanism used in T ans o me s is called "Scaled Do -P oduc
A en ion." The inpu consis s o que ies (𝑄), keys (𝐾), and alues (𝑉) ma ices. The
dimensions o he que ies and keys a e 𝑑𝑘 , and he dimension o he alues is 𝑑𝑣. The
p ocess can be summa ized as ollows:
1. Do P oduc s: Compu e he do p oduc s o he que y wi h all keys.
2. Scaling: Di ide each do p oduc by he squa e oo o he dimension o he keys.
3. So max: Apply he so max unc ion o ob ain he weigh s o he alues.
4. Weigh ed Sum: Compu e he ou pu as he weigh ed sum o he alues.
The ma hema ical o mula is:
𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(𝑄,𝐾,𝑉)=𝑠𝑜𝑓𝑡𝑚𝑎𝑥(𝑄𝐾𝑇
√𝑑𝑘)𝑉
The e a e wo commonly used a en ion mechanisms: addi i e a en ion and do -p oduc
(mul iplica i e) a en ion. Addi i e a en ion compu es he compa ibili y unc ion using a
eed- o wa d ne wo k wi h a single hidden laye , while do -p oduc a en ion di ec ly uses
he do p oduc s o que ies and keys. Do -p oduc a en ion, especially when scaled, is as e
and mo e space-e icien as i le e ages op imized ma ix mul iplica ion code. Fo small
alues o 𝑑𝑘, bo h mechanisms pe o m simila ly. Howe e , o la ge alues o 𝑑𝑘, scaling
6
he do p oduc s by √𝑑𝑘 is necessa y o keep he so max g adien s manageable and he
model ainable.
2.4. MULTI-HEAD ATTENTION
Mul i-head a en ion is a undamen al componen in ans o me models. This mechanism
ex ends he concep o single-head a en ion by employing mul iple a en ion heads o
p ocess he inpu da a in pa allel, each head a ending o di e en aspec s o he inpu
simul aneously.
The p ima y idea is o p ojec he que ies (𝑄), keys (𝐾), and alues (𝑉) linea ly in o mul iple
subspaces, each o educed dimensionali y. This is achie ed h ough di e en lea ned linea
ans o ma ions, c ea ing mul iple se s o Q, K, and V o each head. Speci ically, he que ies,
keys, and alues a e p ojec ed in o 𝑑𝑞, 𝑑𝑘, and 𝑑𝑣 dimensions, espec i ely. These
p ojec ions a e hen p ocessed h ough he a en ion unc ion concu en ly. The ou pu
om each head is a 𝑑𝑣 - dimensional ec o , and hese ou pu s a e conca ena ed and
linea ly ans o med o p oduce he inal ou pu . The o mula o his ope a ion is as ollows:
𝑀𝑢𝑙𝑡ℎ𝑒𝑎𝑑(𝑄,𝐾,𝑉)𝐶𝑜𝑛𝑐𝑎𝑡=(ℎ𝑒𝑎𝑑1,…,ℎ𝑒𝑎𝑑ℎ)𝑊𝑂 , whe e:
ℎ𝑒𝑎𝑑𝑖=𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(𝑄𝑊𝑄𝑖,𝑘𝑊𝐾𝑖,𝑉𝑊𝑉𝑖)
Mul i-head a en ion allows he model o a end o in o ma ion om di e en
ep esen a ion subspaces a a ious posi ions. This is pa icula ly bene icial as i enables he
model o cap u e iche ela ionships wi hin he da a compa ed o single-head a en ion,
which may a e age ou sub le bu c ucial de ails.
Fo example, in a model wi h ℎ=8 heads, i he model dimensionali y (𝑑𝑚𝑜𝑑𝑒𝑙) is 512, each
head ope a es in a subspace whe e 𝑑𝑞=𝑑𝑘=𝑑𝑣=𝑑𝑚𝑜𝑑𝑒𝑙
ℎ=64. Despi e he inc ease in he
numbe o a en ion heads, he compu a ional cos emains manageable due o he educed
dimensionali y pe head.
Types o Mul i-Head A en ion:
• In sel -a en ion, he que ies, keys, and alues a e de i ed om he same inpu
sequence, 𝑄=𝐾=𝑉=𝑋. This allows he model o cap u e dependencies wi hin
he same sequence.
• Used in he T ans o me decode , masked sel -a en ion ensu es ha he model
does no a end o u u e posi ions in he sequence. This is achie ed by applying a
mask o he a en ion ma ix, se ing illegal posi ions o −∞.
• In c oss-a en ion, he que ies come om he p e ious decode laye , while he keys
and alues come om he encode 's ou pu . This mechanism enables he decode o
a end o di e en pa s o he inpu sequence, acili a ing mo e e ec i e
in o ma ion in eg a ion om he encode .
7
3. METHODOLOGY
This pape will ollow he PRISMA (P e e ed Repo ing I ems o Sys ema ic Re iews and
Me a-Analyses) guidelines o ensu e a comp ehensi e and anspa en li e a u e e iew
p ocess.
A comp ehensi e sea ch s a egy was implemen ed ac oss se e al academic da abases and
eposi o ies o iden i y ele an publica ions. The sea ched da abases included IEEE Xplo e,
Scopus, Web o Science, ScienceDi ec , Seman ic Schola , and Google Schola o b oade
each. Rele an Gi Hub eposi o ies we e also e iewed o addi ional esou ces.
The sea ch was conduc ed by sea ching he ollowing keywo ds in he ollowing sou ces:
▪ IEEE Xplo e, Web o Science, and Scopus: " ans o me " AND " ime se ies" AND
" o ecas " in he sea ch ields: Ti le, Abs ac , o Keywo ds.
▪ Seman ic Schola and Google Schola : " ans o me ", " ime se ies", " o ecas "
▪ Gi Hub Reposi o ies: Key eposi o ies sea ched included:
o Awesome Time Se ies Fo ecas ing/P edic ion Pape s (ddz16, 2022/2024)
o T ans o me s in Time Se ies Re iew (Wen, 2022/2024b)
o AI o Time Se ies (AI4TS) Pape s, Tu o ials, and Su eys (Wen, 2022/2024a)
o Time Se ies AI Pape s (xiyuanzh, 2021/2024)
Inclusion and Exclusion C i e ia - Publica ions we e selec ed based on he ollowing c i e ia:
▪ Language: Only pape s w i en in English we e conside ed.
▪ Con en : The pape mus speci ically add ess ime se ies o ecas ing - no any o he
kind o da a no o he a ea ela ed o ime se ies.
▪ Publica ion S a us: Only pee - e iewed a icles published in epu able sou ces we e
included.
▪ Accessibili y: Documen s mus be accessible wi h a No a IMS s uden ID.
A minimum numbe o ci a ions was no used as a c i e ion since his a ea is e y ecen and
mos o he pape s ha e been published in he las 2 yea s, so a small numbe o ci a ions
does no ha e a lo o meaning. How he pape s we e selec ed is shown in Figu e 3.1.
8
Figu e 3.1 – Schema ic Rep esen a ion o he Pape s’ Selec ion P ocess
O he 11 pape s no p esen in Web o Science no Scopus bu pee - e iewed – eigh we e
accep ed as con e ence pos e s and he o he h ee we e published in jou nals. These we e
accep ed as hey all a e pee - e iewed, and one o hese cases is he pape on Pa chTST,
which is ci ed equen ly in la e pape s.
The e we e h ee main easons why ha ini ially passed he selec ion phase we e
subsequen ly ejec ed upon close inspec ion: Only he p ep in was a ailable as seen wi h
pape s by Eisenach e al. (2022), and Ni on e al. (2023); The ocus was on spa io- empo al
o ecas ing a he han ime-se ies o ecas ing, as in he wo k by Liang e al. (2022); The
model p oposed was no a ans o me , exempli ied by he s udy om Z. Gong e al. (2023).
Fo he Da a Collec ion P ocess, da a om each pape we e me iculously ca alogued
manually in o an Excel shee . This allowed o e i ica ion and summa iza ion while eading
and anno a ing he documen s. Reco ded da a ields include:
• In he excel shee “in o” - T ans o me name, pape i le, URL, publica ion da e,
sea ch engine, hos ing websi e, documen ype, e en name, e en da e, keywo ds, i
pee e iew is public, au ho s and a ilia ions, coun y o o igin o unding, unding
ype (public, p i a e, o bo h), supplemen a y ma e ial exis ence, numbe o ci a ions
in Web o Science and Scopus, page coun , annex page coun , e e ence coun , code
a ailabili y, Uni a ia e o Mul i a ia e o ecas ing, I e a i e s di ec mul i-s ep
o ecas ing, da ase s used, da ase links, da ase desc ip ions, o ecas ing ho izons,
me ics, loss unc ions, posi ional encoding, p ep ocessing s eps, aining and uning
9
de ails, a en ion mechanisms, c i ique o p io wo k, con ibu ions, u u e esea ch
di ec ions, and addi ional no es. Emp y cells «we e illed wi h "-".
• In he excel shee “agglome a ion” – Agglome a ion o he esul s o di e en pape s
by me ics and o ecas ing ho izons.
P ima y ou comes: To iden i y and analyse he eme ging ends, inno a ions, and
me hodological ad ancemen s in ans o me models ailo ed o ime se ies o ecas ing; To
pinpoin exis ing gaps, challenges, and limi a ions in he scien i ic knowledge and o s a e-o -
he-a ans o me models; To p opose u u e esea ch di ec ions and po en ial
echnological ad ancemen s.
Seconda y ou comes: Compile o he mos commonly used da ase s and e alua ion me ics
in he li e a u e, p o iding a s anda dised e e ence o u u e esea ch. Analyse publica ion
ends, including he numbe o pape s published o e ime, he mos common publica ion
enues, and he geog aphical dis ibu ion o esea ch e o s. Asses he exis ence o bias. To
de e mine he ex en o he impac o he inno a ions. P o iding a de ailed imeline
ou lining he key changes p oposed in each pape as well as hei mo i a ion.
To assess he isk o bias, he unding sou ces o each pape we e e iewed, no ing whe he
hey we e disclosed and iden i ying he na u e (public, p i a e, o mixed) and o igin
(coun y) o he unding. Nex , he s a ed biases we e ex ac ed om he pape s, speci ically
looking o decla a ions o po en ial con lic s o in e es by he au ho s. To assess
publica ion bias, he pe o mance o he ans o me s was analysed, compa ing he esul s
o he pape s ha used MSE as he me ic in ch onological o de , o see i he e was a
endency o only imp o ing models. Addi ionally, he use o da ase s was e alua ed, no ing
he di e si y, accessibili y, and geog aphic o igin o he da ase s used in he s udies.
Cha GPT was used as an edi o o de ec ypos and imp o e he English used, ensu ing
cla i y and sui abili y o a hesis.
10
4. PAPERS’ ANALYSIS
This sec ion consis s o s udying he 99 Pape s’ cha ac e is ics.
4.1. DATES, DATABASES AND CONFERENCES
The da e he pape s a e made a ailable, and he da es hey a e published a e o en
di e en . Some imes hey a e made public be o e publishing h ough A xi o Open e iew,
which a e p e-p in s be o e pee - e iew. Some imes, hey a e only made public a e being
p esen ed a con e ences. The da es a e he same i hey a e published as pa o a book.
The da es o publica ion pos -pee - e iew we e used o he plo s.
Figu e 4.1 – Numbe o Pape s Published pe Yea and pe Mon h by Publishe - Excluding
2024
The yea ly dis ibu ion shows ha he numbe o pape s pe yea has inc eased
exponen ially. The mos common sou ce is IEEEExplo e, wi h A xi , Else ie , MDPI and
Sp inge also ha ing a signi ican numbe o pape s. F om he mon hly dis ibu ion, i is clea
ha some websi es ha e only accep ed one o wo pape s, such as Js age, epubs, Neu IPS,
AAAI, and ACM. I is also shown ha some websi es publish pape s h oughou he yea ,
such as IEEEExplo e, A xi , MDPI, and Sp inge . Decembe is also he mon h wi h he mos
pape s published and he mon hs wi h he leas publica ions a e Janua y, Ma ch, and
Sep embe .
Figu e 4.2 – P esence o pape s in Web o Science and Scopus
11
Al hough some pape s a e only p esen in Scopus, none a e only on Web o Science.
Figu e 4.3 – Type o pape s
A icles, which a e pee - e iewed pieces published in jou nals, cons i u e he mos
signi ican po ion a 38.7%. Con e ence pape s, which a e p esen ed a con e ences, make
up 33.8%. P oceedings pape s, which a e included in he o icial eco d o he con e ence,
accoun o 27.5%. This indica es a well- ounded ep esen a ion o academic esea ch
dissemina ion me hods.
4.2. CITATIONS
The e a e 10 pape s wi h no ci a ions on ei he Web o Science no Scopus, and ano he 13
pape s ha a en’ p esen in Web o Science and ha e 0 ci a ions on Scopus. This is o be
expec ed due o he ecency o a lo o he pape s.
Figu e 4.4 shows he numbe o pape s pe numbe o binned ci a ions.
18
Figu e 4.8 – A e age MSE pe ans o me , o de ed by publica ion da e
The boxplo s in blue mean ha he pape s use all o he 9 mos common da ase s, while in
ed hey only use a subse o hem. I is isible ha he e a e ans o me s ha achie e
wo se esul s han p e ious models, sugges ing he e is a ela i ely low publica ion bias,
which could mean ha he main goal o some pape s is o explo e how well di e en
s a egies wo k, and no necessa ily on only imp o ing accu acy.
Howe e , he p esence o bias canno be en i ely dismissed. O he 100 pape s analysed,
he e a e 62 unique da ase s, many o which a e no publicly a ailable. This lack o
accessibili y and he weak compa isons made - o en using only a single da ase o
compa ing o a limi ed numbe o ans o me s - sugges ha some deg ee o bias may exis
as he pe o mance o hose models isn’ obus ly assessed. Fu he mo e, some pape s
employ me ics ha a e no widely used, complica ing compa isons wi h o he s udies in he
li e a u e.
Addi ionally, he esul s o he ans o me s ha e some inhe i andomness, as seen in Table
0.1 – E olu ion o In o me me ics h oughou di e en pape s, bu only h ee pape s
show he s anda d de ia ion o he esul s. This could mean ha he e is some skewness in
he esul s as he esul s shown a e p esumably he bes esul s ob ained by he au ho s.
19
4.7. FORECASTING TRENDS
This sec ion ocuses on inding o ecas ing ends by plo ing he e olu ion o he numbe o
di e en ypes o o ecas ing s a egies.
Figu e 4.9 – Cumula i e numbe o pape s by o ecas ing ype
Ini ially, esea ch on ans o me s in ime se ies o ecas ing explo ed bo h mul i a ia e and
uni a ia e condi ions. Mul i a ia e o ecas ing in ol es p edic ing mul iple ime se ies
simul aneously, cap u ing hei in e dependencies, while uni a ia e o ecas ing deals wi h
p edic ing a single ime se ies based solely on i s pas alues.
O e ime, howe e , he e has been a no iceable shi owa d p ima ily ocusing on
mul i a ia e scena ios. Despi e some pape s claiming ha he p oposed ans o me s a e
sui able o bo h se ings, demons a ions a e p edominan ly con ined o mul i a ia e da a.
This end highligh s a g owing in e es in le e aging he complex in e dependencies wi hin
mul i a ia e da ase s, which o en encapsula e iche in o ma ion and mo e ealis ic
condi ions o o ecas ing.
Figu e 4.10 – Cumula i e numbe o pape s by o ecas ing ype
20
In he ea ly s ages, ans o me s in ime se ies o ecas ing p ima ily employed an i e a i e
app oach, whe e p edic ions we e made one s ep a a ime, using he ou pu o he p e ious
s ep as inpu o he nex p edic ion. Howe e , by he end o 2022, he e was a signi ican
shi owa ds using di ec mul i-s ep o ecas ing, whe ein models p edic mul iple s eps
ahead in a single o wa d pass. This change was d i en by he need o add ess e o
accumula ion inhe en in he i e a i e app oach.
4.8. PREPROCESSING, SPLITS, AND TUNING
The sys ema ic li e a u e e iew e eals consis en ends in p ep ocessing echniques
ac oss a ious da ase s, ensu ing da a quali y and model compa ibili y. No maliza ion,
pa icula ly z-sco e s anda diza ion, is p ominen ly employed o achie e ze o mean and uni
a iance. This me hod is applied ac oss di e se da ase s, including elec ici y, a ic, e ail,
and ola ili y da a, s abilizing he da a and enhancing model pe o mance. Log
ans o ma ion is ano he common p ac ice, pa icula ly o da ase s wi h skewed
dis ibu ions such as e ail sales and ola ili y, helping o s abilize a iance and no malize
da a dis ibu ion. These p ep ocessing s eps a e c i ical o main aining model s abili y and
mi iga ing issues a ising om di e ing da a scales.
Handling missing alues is a signi ican aspec o p ep ocessing, wi h a ious me hods
employed o add ess gaps in he da a. The e ail da ase , o ins ance, esamples da a a
egula daily in e als and impu es missing days using he las a ailable obse a ion.
Simila ly, Bi coin da a unde goes igo ous da a cleaning o emo e missing alues and
inconsis en da a o ma s. O he echniques include K-Nea es Neighbou (KNN) da a
impu a ion and mean illing algo i hms. These me hods ensu e ha he da ase s a e
comple e and eliable, which is c ucial o e ec i e model aining and accu a e p edic ions.
Fea u e ans o ma ion and ex ac ion a e also p e alen p ep ocessing s eps, enhancing
he model's abili y o cap u e pa e ns and ends in he da a. Da ase s such as elec ici y
and a ic include day-o -week, hou -o -day, and ime index as eal- alued inpu s, ea ing
ca ego ical a iables like en i y iden i ie s sepa a ely. Ad anced echniques like Fas Fou ie
T ans o m (FFT) and seasonal- end decomposi ion a e used o ex ac in o ma i e ea u es
and smoo h he da a. Addi ionally, segmen a ion in o pa ches, da a chunking, and sampling-
ype coding a e applied o s abilize ime se ies da a and acili a e e ec i e model aining.
These p ep ocessing echniques collec i ely con ibu e o he obus ness and accu acy o
he models ac oss a ious da ase s. Decomposi ion echniques in eg a ed in he ans o me
a chi ec u e a e s udied in sec ion 5.1.4 Decomposi ion.
The e iewed pape s consis en ly spli da ase s in o aining, alida ion, and es ing se s,
al hough he speci ic a ios a y. A common app oach is o use a 70/15/15 o 70/10/20 spli ,
ensu ing ha a subs an ial po ion o he da a is alloca ed o aining while ese ing
smalle pa s o alida ion and es ing o e alua e model pe o mance. Fo ins ance, he ETT
21
da ase equen ly uses a 6:2:2 a io o aining, alida ion, and es ing, whe eas o he
da ase s like a ic and elec ici y o en employ a 7:1:2 spli . Some s udies also ollow a
ime-based spli ing me hod, such as di iding da a in o speci ic mon hs o aining,
alida ion, and es ing, exempli ied by he ECL da ase 's 15/3/4 mon h spli and he wea he
da ase 's 28/10/10 mon h con igu a ion. These consis en ye lexible s a egies ac oss
di e en da ase s highligh he emphasis on obus model alida ion and es ing o ensu e
gene alizabili y and accu acy.
The uning o pa ame e s in he e iewed pape s in ol es a combina ion o andom sea ch,
g id sea ch, and au oma ed hype pa ame e op imiza ion echniques. G id sea ch and
andom sea ch a e commonly used o explo e a ious combina ions o hype pa ame e s
such as lea ning a e, d opou a e, ba ch size, numbe o laye s, and a en ion heads. Fo
ins ance, some s udies conduc g id sea ches o e speci ic lea ning a es, α- alues, encode
and decode blocks, while o he s use andom sea ch o iden i y he bes se ings ac oss
mul iple i e a ions, such as 200 o he Exchange Ra e and Elec ici y da ase s.
Mo e ad anced me hods, like hose implemen ed using he Op una amewo k, in ol e
au oma ed hype pa ame e op imiza ion. Op una uses e icien sampling and p uning
me hods, like he ee-s uc u ed Pa zen es ima o and he asynch onous successi e hal ing
algo i hm, o i e a i ely e ine he sea ch space based on his o ical pe o mance da a. This
app oach signi ican ly educes he manual e o and imp o es he e iciency o inding
op imal hype pa ame e alues. These uning p ac ices ensu e obus and well-op imized
models o a ious o ecas ing asks.
22
5. EVOLUTION AND DISCUSSION
T ans o me s ha e signi ican ly e ol ed o add ess he unique challenges o ime se ies
o ecas ing, wi h a ious modi ica ions enhancing hei capabili y o cap u e empo al
dynamics and imp o e p edic i e accu acy. This sec ion del es in o key a eas o hese
ad ancemen s, ocusing on each di e en componen and aspec o he ans o me s.
The o ganiza ion o his sec ion is shown in Figu e 5.1. I di ides he ans o me in o “Inpu
Rep esen a ion”, “Modelling”, “Model Op imiza ion”, “Types o Lea ning” and “Repea ed
T ans o me s”. The i s h ee a e u he sub-di ided in o hei composing elemen s, and
each o hese subsec ions g oups he mechanisms in oduced by hei simila i y.
Figu e 5.1 – O ganiza ion o he “E olu ion and Discussion” Sec ion
This amewo k was chosen o be e showcase how each aspec o he ans o me model
has e ol ed, making i easie o ind pa e ns, and allowing he eade o ocus on one
aspec a a ime. The e is some o e lap, o epe i ion, o some inno a ions in he di e en
sec ions as some all unde mo e han one ca ego y.
In o de o e alua e he Ce ain y o E idence and impac o each ou come, he abla ion
s udies p esen in he pape s, whe e he au ho s emo e one o he in oduced componen s
one by one o see how each a ec s he pe o mance o he model, we e analysed. The las
pa ag aph o each ou come is a s udy o he ce ain y o e idence by gi ing an o e all
summa y o he ele an abla ion s udies.
23
The summa y o he inno a ions in oduced by each pape is ch onologically o de ed in
Appendix C (Table o Summa y o Changes).The e olu ion o he complexi y equa ions and
he di e en app oaches o each pape a e in Appendix A (Complexi y Equa ions).
5.1. INPUT REPRESENTATION
A. Zeng e al. (2022) c i ically examine he e ec i eness o T ans o me -based models o
long- e m ime se ies o ecas ing (LTSF). Despi e hei popula i y, he au ho s a gue ha
T ans o me s su e om empo al in o ma ion loss due o hei pe mu a ion-in a ian sel -
a en ion mechanism. They in oduce a simple one-laye linea model, LTSF-Linea , o
compa ison and demons a e h ough ex ensi e expe imen s on nine eal-wo ld da ase s
ha his model consis en ly ou pe o ms T ans o me -based solu ions. The s udy e eals
ha he empo al modelling capabili ies o T ans o me s un il hen we e o e s a ed o
cu en benchma ks, sugges ing he need o new esea ch di ec ions and a e-e alua ion o
T ans o me -based app oaches o ime se ies analysis asks.
Following his s udy, subsequen esea ch has inc easingly ocused on enhancing inpu
ep esen a ion o be e p ese e he o de o inpu s and mo e e ec i ely cap u e con ex
and empo al dependencies. This sub-sec ion co e s a ious me hods o achie e his, as
illus a ed in Figu e 5.2. These me hods include imp o ing posi ional encoding, u ilizing
co a ia es, segmen ing he da a, and decomposing he da a.
Figu e 5.2 – Di ision and Sub-di ision o he Inpu Rep esen a ion, Showing he Numbe o
Models ha all unde each Sub-di ision
24
5.1.1. Posi ional Embedding
T adi ional T ans o me s use sinusoidal posi ional encodings o e ain sequence o de , as
sel -a en ion mechanisms a e inhe en ly o de -agnos ic. Va ious s udies ha e modi ied
hese me hods o enhance pe o mance in ime se ies o ecas ing by cap u ing complex
pa e ns and add essing unique empo al challenges.
5.1.1.1. Adap i e and Lea nable Embeddings
Some models in eg a e adap i e and lea nable embeddings, allowing he model o
dynamically adjus o di e en da ase s, educing manual uning and op imizing empo al
ea u e ep esen a ion du ing aining. Zhong e al. (2023), in NT o me , u ilize lea nable
imes amp encoding o e ec i ely handle he empo al ea u es. This me hod in ol es
con e ing empo al ea u es - seconds, minu es, and hou s - in o dense ec o s h ough an
embedding laye ha is lea ned du ing aining, imp o ing sequence unde s anding and
p edic ion based on his o ical da a. Y. Chen e al. (2023), in JTFT, use a lea nable posi ion
embedding ha is added o he join ime- equency domain ep esen a ion, ensu ing he
model cap u es he sequen ial o de o he da a in he combined Time Domain and
F equency Domain space.
Yang & Lu (2023), in Fo e o me , in oduce Adap i e Posi ional Encoding, combining linea
𝛼𝑖𝑡+𝑖𝛽 and sinusoidal sin(𝛼+𝑖𝛽) componen s o cap u e non-pe iodic and pe iodic
pa e ns, espec i ely. The lea nable pa ame e s 𝛼𝑖 and 𝛽𝑖 allow adap a ion o speci ic
empo al dynamics. Simila ly, Ni e al. (2024), in Basis o me , in oduce lea nable and
in e p e able basis ec o s h ough adap i e sel -supe ised lea ning. This app oach ea s
he his o ical and u u e sec ions o a ime se ies as wo dis inc iews, using con as i e
lea ning o es ablish and e ine hese basis ec o s. The lea ned bases eplace he adi ional
posi ional encoding mechanism, embedding bo h he posi ion and in insic ea u es o he
ime se ies wi hin he model's a chi ec u e
Fu he ex ending hese ideas, Z. Zhang e al. (2024), in SageFo me , add lea nable okens o
he inpu sequence o each se ies. These okens cap u e o e a ching pa e ns and
dependencies ac oss di e en se ies. Du ing he encoding p ocess, hey in e ac wi h he
se ies da a h ough i e a i e message passing, in ol ing in a-se ies in e ac ions ia s anda d
ans o me mechanisms and in e -se ies in e ac ions ia g aph neu al ne wo ks (GNNs).
5.1.1.2. Mul i-Resolu ion Embeddings
Inco po a ing mul i- esolu ion embeddings in ime se ies o ecas ing enhances he abili y o
cap u e pa e ns a di e en empo al scales. Y. Cao & Zhao (2023), in DWT o me , use 2D-
Fea u esBlock ha enhances he inpu ep esen a ion o ime se ies da a by le e aging
con olu ional neu al ne wo ks. A e he wa ele decomposi ion ans o ms he 1D ime
se ies in o 2D enso s, he 2D-Fea u esBlock applies an Incep ion-like s uc u e wi h mul i-
scale con olu ional il e s o cap u e bo h local and global pa e ns. This mul i-scale
25
con olu ional app oach e ec i ely ex ac s complex in a-pe iod and in e -pe iod a ia ions.
Simila ly, G. Tong e al. (2024), in RSM o me , al e he inpu ep esen a ion by using a
mul iscale app oach. I employs up-and-down sampling echniques o ans o m he inpu
da a in o mul iple esolu ions, cap u ing empo al dependencies a di e en g anula i ies.
This enables he model o e ec i ely manage long- ange dependencies and combine
in o ma ion om a ious scales.
In ano he app oach, X. Wang, Liu, Du, e al. (2023), in CL o me , al e he inpu
ep esen a ion by i s applying dila ed causal con olu ions o cap u e mul i-scale empo al
pa e ns, ollowed by sequence decomposi ion o sepa a e he ime se ies in o seasonal and
end componen s. This p ep ocessing e ines he inpu da a, emphasizing egula and
p edic able pa e ns. The inpu ep esen a ion is u he e ined by segmen ing he ime
se ies and applying using he locally g ouped au oco ela ion (LGAC) mechanism ha
calcula es au oco ela ion wi hin hese segmen s, enhancing he ocus on local empo al
dynamics. Adding o hese me hodologies, Y. Li, Lu, e al. (2023), in Con o me , u ilize an
ad anced inpu ep esen a ion. I employs Fas Fou ie T ans o m (FFT) o compu e
co ela ions among a iables, emphasizing ele an ea u es wi h a so max unc ion.
Addi ionally, he model ep esen s he da a a a ious empo al esolu ions, embedding
hese mul i-scale ep esen a ions in o a la en space. These embeddings a e conca ena ed
and weigh ed o o m a comp ehensi e inpu ha cap u es in ica e empo al pa e ns.
Y. Zhang, Wu, e al. (2024), in MTPNe , in oduce a dimension-in a ian (DI) embedding
echnique, which, u ilizing a 1-laye CNN, p ojec s mul i a ia e ime se ies da a in o a highe -
dimensional space while p ese ing bo h spa ial and empo al dimensions. The da a is hen
decomposed in o seasonal and end componen s. The seasonal componen is segmen ed
in o non-o e lapping pa ches o a ying sizes co esponding o di e en empo al scales.
Each le el o he hie a chical py amid s uc u e p ocesses hese pa ches, allowing he model
o cap u e and lea n om ine o coa se empo al dependencies. The end componen is
handled sepa a ely using a linea laye .
Dai e al. (2023), in PDF, ans o m he inpu ep esen a ion om a 1D ime se ies in o a
iche 2D o ma . This in ol es decoupling he se ies in o sho - e m and long- e m
componen s using he Fas Fou ie T ans o m (FFT), hen eshaping hese componen s in o
2D enso s. Rows in hese enso s cap u e sho - e m a ia ions, while columns cap u e
long- e m a ia ions. These enso s a e u he di ided in o pa ches and slices o sepa a ely
model long- e m and sho - e m empo al pa e ns. The Dual Va ia ions Modeling Block
(DVMB) hen p ocesses hese pa ches and slices o e ec i ely cap u e and model bo h
sho - e m and long- e m a ia ions.
Cen & Lim (2024), in Pa chTCN-TST, employ TCN o embedding, using causal con olu ions o
p ese e empo al o de . The TCN block ensu es ha he embedding does no use u u e
da a, only ocusing on pas and p esen , e ec i ely embedding he empo al in o ma ion
di ec ly in o he da a ep esen a ion. Addi ionally, he model employs a pa ching ope a ion
26
ha segmen s he inpu sequence in o smalle sub-se ies o pa ches. This segmen a ion
allows he model o ocus on local dependencies wi hin each pa ch, imp o ing he
g anula i y o ea u e ex ac ion and educing compu a ional complexi y.
5.1.1.3. Tempo al Focused Embeddings
Focusing on empo al-speci ic embeddings signi ican ly enhance he model's abili y o
cap u e and p edic ime se ies da a by e ining how posi ional in o ma ion is ep esen ed.
Y. Liu e al. (2024), in iT ans o me , eplace adi ional posi ional encoding by es uc u ing
he da a o ocus on a ia e okens a he han empo al okens. In his app oach, each
a iable in he ime se ies is ea ed as a sepa a e oken, and he a en ion mechanism is
applied ac oss hese a ia e okens o cap u e hei in e dependencies. This in e sion o
da a dimensions inhe en ly inco po a es he o de and ela ionship wi hin each a ia e's
ime se ies h ough he s uc u e o he da a and he model's a chi ec u e, making explici
posi ional encoding unnecessa y.
Simila ly, Y. Li, Qi, e al. (2023), in SMART o me , use Time-Independen Embedding (TIE) o
decouple posi ional encoding om alue embeddings. TIE uses h ee sepa a e lea nable
p ojec ion ma ices o ep esen minu e/hou , weekday, and mon h, ensu ing ha
posi ional in o ma ion is added wi hou dis o ing he ac ual da a alues.
Domínguez-Cid e al. (2023), in TEFNEN, eplace he s anda d embedding laye s o he
anilla ans o me wi h ully connec ed single-laye neu al ne wo ks. Addi ionally, Ins ead
o using he s anda d posi ional encoding o p o ide in o ma ion abou he o de o he
sequence, TEFNEN inco po a es a Recu en Neu al Ne wo k (RNN) laye wi hin he encode -
decode s uc u e. This RNN laye cap u es he sequen ial na u e and empo al
dependencies o ime-se ies da a.
Y. Zhang, Ma, e al. (2024), in MTST, employ ela i e posi ional encoding (RPE) ha cap u es
he ela i e dis ances be ween okens, ocusing on hei in e als a he han absolu e
posi ions. This me hod enhances he model's abili y o ecognize and p edic empo al
dependencies and pe iodic pa e ns in ime-se ies da a. By encoding he di e ences
be ween oken posi ions, MTST e ec i ely lea ns ela ionships ha occu a a ying
in e als.
In ano he app oach, Z. Wang e al. (2024), in PWD o me , inco po a e posi ion weigh s,
which enhance he model's sensi i i y o he o de o ime se ies da a. These weigh s assign
a ying impo ance o di e en posi ions wi hin he inpu sequence.
Finally, Zeng e al. (2023), in Se o me , in oduce Bina y Posi ion Encoding, wi h in ablock
and in e block posi ion encodings. The in ablock posi ion encoding handles posi ions
segmen s he sequence in o smalle blocks and uses a simpli ied, disc e ized ODE app oach
o e icien ly cap u e local posi ional dynamics, educing compu a ional complexi y.
In e block posi ion encoding hen ecu si ely manages he ela ionships be ween hese
27
blocks o manage long- ange dependencies and p e en issues like baseline d i in longe
sequences.
5.1.1.4. Adding o Con ex ual In o ma ion
Shabani e al. (2023), in Scale o me , al e he inpu embedding by inco po a ing h ee
componen s: alue, empo al, and posi ional embeddings. The alue embedding includes an
addi ional indica o o he sou ce o obse a ions (look-back, ze o ini ializa ion, o p e ious
p edic ions). The empo al embedding maps ime s amps o he hidden dimension, adjus ed
by he cu en scale ac o . The posi ional embedding is adap ed o he di e en scales si.
These changes enable he model o e ec i ely in eg a e and p ocess mul i-scale
in o ma ion, imp o ing o ecas ing accu acy. Building on his idea, Qu e al. (2024), in
Fo wa d o me , change he inpu ep esen a ion by inco po a ing a comp ehensi e
embedding s uc u e ha includes alue embeddings o daily load da a, imes amp
embeddings o empo al con ex (day, week, and mon h), and in o ma ion embeddings o
addi ional con ex ual ac o s such as holidays, empe a u e, and p essu e. This en iched
inpu ep esen a ion allows he model o cap u e and u ilize he in ica e empo al and
con ex ual dependencies in he load da a.
Simila ly, D ouin e al. (2022), in TACTiS, inco po a e imes amps, enabling he model o
manage i egula sampling in e als and main ain accu a e empo al o de , s a ic and ime-
a ying co a ia es, and Boolean masks o missing alues in he inpu , en iching con ex o
o ecas ing. Posi ional encodings e lec ime-speci ic ac o s like seasonali y, enhancing
empo al dynamics unde s anding. Fu he ex ending his app oach, Ashok e al. (2024), in
TACTiS-2, in oduce a dual-encode a chi ec u e, p ocessing p ima y ime se ies da a and
co a ia es wi h posi ional encodings. One encode handles ma ginal dis ibu ions, while he
o he handles copula dis ibu ions, specializing in di e en aspec s o he inpu da a.
5.1.1.5. Specialized Embeddings
C. Zhang e al. (2023), in Causal o me , modi y he inpu ep esen a ion by combining
empo al and causal ea u es. I uses a Time Encode wi h P obSpa se Sel -a en ion o
cap u e empo al dependencies and a Causal Encode o iden i y causal ela ionships among
a iables h ough a G ange causali y g aph. This causal in o ma ion, embedded as a spa se
adjacency ma ix, is in eg a ed in o he inpu da a, which is hen p ocessed by a gene a i e
mul i-head decode .
Jawed & Schmid -Thieme (2022), in GQFo me , in oduce unique ime-se ies ID embedding
ha assigns each ime se ies a dis inc iden i ie , which is embedded in o a high-dimensional
space and in eg a ed wi h he inpu da a. This allows he model o lea n and di e en ia e
pa e ns speci ic o each ime se ies, imp o ing i s accu acy ac oss di e se da ase s.
Addi ionally, he implici quan ile le el embedding samples quan ile le els om a uni o m
dis ibu ion and embeds hese in o a high-dimensional space. These embeddings a e
34
ealigned o main ain he o iginal sequence o de , in eg a ing mul i-scale empo al pa e ns
o enhanced o ecas ing.
5.1.3.4. Discussion
Segmen a ion s a egies in ime se ies o ecas ing ha e e ol ed om basic olling windows
and simple pa ches o mo e adap i e and dynamic me hods, demons a ing e sa ile
echniques ha enhance o ecas ing pe o mance. Pa ch-based mechanisms e ec i ely
cap u e local pa e ns, educe compu a ional load, and manage long ime se ies e icien ly
by ea ing segmen s as dis inc uni s. Window-based segmen a ion ensu es models can
con inuously e alua e and adap o changing condi ions, while adap i e segmen a ion allows
o de ailed analysis o empo al dynamics and dependencies. These me hods unde sco e a
shi owa ds mo e con ex -awa e and g anula da a p ocessing.
Resea ch on his has eached a poin whe e i indica es ha pa ch size con ols he abili y o
ans o me s o lea n empo al pa e ns a di e en equencies, wi h sho e pa ches being
e ec i e o localized, high- equency pa e ns and longe pa ches needed o mining long-
e m seasonali ies and ends. (Y. Zhang, Ma, e al., 2024)
Rega ding he ce ain y o e idence, while some models lack dedica ed abla ion s udies, he
consis en indings ac oss he o he models p o ide s ong e idence ha segmen a ion
s a egies a e p omising o enhancing ans o me -based ime se ies o ecas ing.
5.1.4. Decomposi ion
Decomposi ion echniques a e c ucial o ime se ies o ecas ing, imp o ing model accu acy
by b eaking down da a in o manageable componen s. These me hods add ess complex
pa e ns, noise, and ends by isola ing end, seasonali y, and esidual componen s,
allowing o p ecise modelling and p edic ion. This enhances model obus ness and
eliabili y, le e aging bo h ime-domain and equency-domain in o ma ion.
5.1.4.1. T end-Seasonali y Decomposi ion
Va ious T ans o me s decompose ime se ies in o end and seasonal componen s o
imp o e o ecas ing accu acy. X. Zhang e al. (2022), in TD o me , use mul iple a e age
il e s o di e en sizes, combined wi h adap i e weigh s, o ex ac end pa e ns. Simila ly,
X. Wang, Liu, Du, e al. (2023), in CL o me , apply dila ed causal con olu ions in he
Tempo al Con olu ion Block (TCB) and hen use sequence decomposi ion o isola e ends
by sub ac ing a mo ing a e age, lea ing he seasonal componen as he emainde . Y.
Huang & Wu (2023), In ADAMS, uses a mo ing a e age o ex ac long- e m ends and
sub ac s his om he o iginal se ies o ob ain he seasonal componen , u he
s a iona izing he se ies by ans o ming i in o a Gaussian dis ibu ion.
35
Y. Zhang, Ma, e al. (2024), in MTPNe , and D. Cao e al. (2024) in TEMPO also u ilize mo ing
a e ages o end ex ac ion. MTPNe sub ac s he end o ge he seasonal componen ,
while TEMPO u he decomposes he se ies in o end, seasonali y, and esiduals using
smoo hing and a e aging echniques. Thwal e al. (2023), in Fede a ed T ans o me , employ
mo ing a e age smoo hing o p ep ocessing, educing noise and enhancing s a iona i y.
5.1.4.2. Decomposi ion Blocks
O he ans o me s sepa a e he inpu da a in o end and seasonal componen s using
specialized decomposi ion blocks. H. Wu e al. (2021), in Au o o me , employ decomposi ion
blocks o ocus on seasonal pa e ns and e ine end p edic ions. In he encode , hese
blocks acili a e he de ec ion o seasonal dependencies, while in he decode , hey
p og essi ely e ine end p edic ions using he end-cyclical componen s. Simila ly, T. Zhou
e al. (2022), in FED o me , use Mix u e o Expe Decomposi ion (MOEDecomp), using a se
o a e age pooling il e s wi h a ying sizes, combined wi h Fou ie and wa ele
ans o ma ions in F equency Enhanced Blocks (FEBs) o p ocess decomposed componen s
in he equency domain.
Ban e al. (2024), in MOEA, enhance he Au o o me wi h MOEDecomp and Wa ele So
Th eshold Denoising (WSTD) o be e ex ac complex ends and educe noise. (Du e al.,
2023), in P e o me , use decomposi ion blocks wi h a Mul i-Scale Segmen -Co ela ion
(MSSC) mechanism in he encode and ecombina ion in he decode . Ouyang e al. (2023a),
in Rank o me , Mul i-Le el Decomposi ion (MLDecomp) wi h Mul iple Mo ing A e age
(MMA) il e s o cap u e ends a di e en scales, using ainable weigh s o e inemen .
Dai e al. (2023), in PDF, inco po a e a mul i-pe iodic decoupling block (MDB) ha models
sho - e m and long- e m a ia ions using ans o me encode laye s and con olu ional
laye s. Las ly, X. Wang, Liu, Yang, e al. (2023), in CN o me , use a se ies decomposi ion
block wi h a e aging and di e encing echniques, e ining seasonal componen s wi h
s acked dila ed con olu ional blocks (SDCBs) and using con olu ional a en ion in he
decode o imp o ed o ecas ing accu acy.
5.1.4.3. F equency Domain T ans o ma ions
Se e al ans o me s u ilize equency domain ans o ma ions o enhance o ecas ing by
cap u ing signi ican pe iodic componen s. Y. Wang, Zhu, & Kang (2023), in DEST o me ,
apply Fas Fou ie T ans o m (FFT) o sepa a e high- equency and low- equency elemen s,
p ocessing he seasonal componen wi h a Mul i-View A en ion mechanism ha conside s
ampli ude and phase o cap u ing complex pe iodic changes. The end componen is
managed wi h a Mul i-Scale A en ion mechanism using con olu ional il e s o a ious
leng hs o cap u e ends a di e en scales. Simila ly, J. Ma & Dan (2023), in FFT-in o me ,
segmen da a, apply FFT o each segmen , and ex ac equency componen s o summa ize
ends and pe iodici ies.
36
Y. Cao & Zhao (2023), in DWT o me , implemen wa ele decomposi ion o c ea e mul i-
esolu ion ep esen a ions, sepa a ing high- equency de ails and low- equency ends.
Con olu ion T ans o me in oduces a F equency Enhanced Block ha maps ime-se ies da a
o he equency domain using Disc e e Fou ie T ans o m (DFT), selec s signi ican
equencies, e ines hem h ough a lea nable ke nel, and con e s he da a back o he ime
domain en iched wi h global ea u es. P. Chen e al. (2023), in Pa h o me , apply DFT o
ex ac signi ican pe iodic componen s and econs uc ing seasonali y wi h In e se DFT. The
esidual da a is p ocessed wi h a e age pooling o ex ac he end, and hese componen s
a e combined and ed in o a mul i-scale ou e o cap u e bo h local and global
dependencies.
S. Huang e al. (2022), in CEEMDAN, use Comple e Ensemble Empi ical Mode Decomposi ion
wi h Adap i e Noise (CEEMDAN) o decompose da a in o mul iple in insic mode unc ions
(IMFs), add essing mode mixing wi h adap i e noise and ca ego izing segmen s by
complexi y using Sample En opy (SE). High-complexi y segmen s a e p ocessed by a
ans o me model, while low-complexi y segmen s a e handled by a Back P opaga ion
Neu al Ne wo k (BPNN). Sasal e al. (2022), in W- ans o me s, u ilize he Maximal O e lap
Disc e e Wa ele T ans o m (MODWT) o decompose ime se ies in o componen s a
di e en scales, applicable o da a o any leng h and p oducing coe icien s ep esen ing
a ious equency bands and ends.
J. Chen e al. (2023), in Seg o me , employ a mul i-componen s decomposi ion app oach
using FFT o cap u e di e en equency pa e ns, decomposing ime se ies in o an o e all
end, odd equency componen s, and e en equency componen s o e ec i ely cap u e
bo h local and global empo al pa e ns.
5.1.4.4. Specialized Decomposi ion
Some ans o me s pe o m Seasonal-T end decomposi ion wi h specialized componen s.
Fan e al. (2023), in TED o me , do i using Loess (STL), and hen applying Tempo al
Con olu ional Ne wo ks (TCNs) o he end and an au oco ela ion mechanism along wi h
STL o he seasonal componen s. The decode p ocesses hese decomposed componen s
using TCN and ea u e usion s a egies.
Simila ly, H. Cao e al. (2023), in InPa o me , employ he E olu iona y Seasonal-T end
Decomposi ion (E oSTD) algo i hm o dynamic bina y decomposi ion, i e a i ely spli ing
ime se ies in o seasonal and end componen s o e ine he model's unde s anding o
unde lying pa e ns. Ouyang e al. (2023b), in STL o me , also use STL Decomposi ion wi h
LOESS (Locally Es ima ed Sca e plo Smoo hing) o sepa a e da a in o seasonal and end-
cyclical componen s. Addi ionally, STL o me inco po a es a Rank Co ela ion Block using
Spea man's Rank Co ela ion o cap u e nonlinea dependencies and ela ionships be ween
anked da a poin s.
37
5.1.4.5. Hyb id Me hods
Some ans o me s in eg a e o he models o decomposi ion and p ep ocessing. Lin e al.
(2021), in SSDNe , embed a ixed- o m s a e space model (SSM) wi hin he T ans o me
a chi ec u e o di ec ly decompose ime se ies da a in o end and seasonali y componen s.
Simila ly, M. Li e al. (2022), in Um o me , employ ea u e decomposi ion and da ase
p ep ocessing using he P ophe algo i hm, which sys ema ically decomposes uni a ia e
ime se ies da a in o end, seasonali y, holiday e ec s, and esidual noise. The decomposed
da a is hen no malized and scaled o imp o e he lea ning e iciency.
J. Tong e al. (2023), in PDT ans, enhance he model's p edic i e abili y by decomposing ime
se ies in o end and seasonali y componen s using con olu ion ope a ions and mul i-laye
pe cep ons (MLPs), espec i ely. This app oach allows he model o sepa a ely model and
p edic dis inc pa e ns.
5.1.4.6. P e-p ocessing and Dual-Domain S a egies
Some ans o me s ocus on ad anced p ep ocessing echniques o enhance model
pe o mance. Y. Liu e al. (2023), in Non-s a iona y T ans o me , employ employ se ies
s a iona iza ion, no malizing ime se ies da a o ze o mean and uni a iance be o e
p ocessing, and hen de-no malizing he p edic ions o main ain in e p e abili y in hei
o iginal con ex . This app oach helps manage he non-s a iona y na u e o eal-wo ld da a.
Simila ly, Zhong e al. (2023), in NT o me , include bo h non- empo al and empo al ea u e
p ocessing. Non- empo al p ocessing emo es i ele an da a, ills missing alues, and
no malizes nume ical da a, while empo al p ocessing aligns ime cha ac e is ics ac oss
di e en s ages, educing ea u e complexi y and empo al misalignmen .
J. Zhu e al. (2023), in SL- ans o me , use he Sa i zky–Golay (SG) il e o smoo h wind
speed da a and he Local Ou lie Fac o (LOF) il e o emo e ou lie s in sola powe
gene a ion da a, enhancing inpu quali y. Yu e al. (2023), in Ds o me , employ a Double
Sampling (DS) Block ha combines downsampling o emphasize long- e m pa e ns and
piecewise sampling o e ain de ailed sho - e m pa e ns, c ea ing ea u e ec o s ha
cap u e bo h mac o and mic o-le el in o ma ion.
Y. Chen e al. (2023), in JTFT, use a join ime- equency domain app oach, ans o ming ime
se ies in o bo h ime-domain (TD) and equency-domain (FD) componen s. They apply a
cus omized Disc e e Cosine T ans o m (CDCT) o ob ain FD componen s, cap u ing essen ial
pe iodic and end elemen s while il e ing ou noise. These FD componen s a e hen
combined wi h ecen TD da a poin s, allowing he model o le e age bo h empo al and
equency in o ma ion.
38
5.1.4.7. Discussion
Decomposi ion echniques in ime se ies o ecas ing a e inc easingly sophis ica ed, ocusing
on isola ing end, seasonali y, and esidual componen s o enhance model accu acy and
obus ness. Recen ends include in eg a ing mo ing a e ages and il e s o undamen al
pa e n ex ac ion, using specialized decomposi ion blocks combined wi h equency domain
echniques o e ined componen sepa a ion, and applying ad anced ans o ma ions like
Fou ie and wa ele o cap u ing complex pe iodic componen s. Dynamic and adap i e
me hods add ess in ica e a ia ions in da a, while hyb id app oaches inco po a e di e se
models and p ep ocessing algo i hms o comp ehensi e decomposi ion. Enhanced
p ep ocessing and dual-domain s a egies emphasize me iculous da a p epa a ion,
le e aging s a iona iza ion, noise educ ion, and combined ime- equency domain analysis
o manage he complexi ies o eal-wo ld da a e ec i ely.
Rega ding he ce ain y o e idence, only 6 o he pape s lack any kind o abla ion s udy wi h
he o he s ongly suppo ing he impac o decomposi ion, demons a ing hei
e ec i eness in enhancing model pe o mance. These s udies consis en ly show ha
ad anced decomposi ion me hods, whe he h ough specialized blocks, equency
ans o ma ions, o combined domain app oaches, signi ican ly imp o e p edic i e accu acy.
The e idence unde sco es ha hese echniques a e c ucial o e ining models' abili ies o
cap u e and p edic complex empo al pa e ns, alida ing hei necessi y and e ec i eness
in he ield.
5.2. MODELING
Recen ad ancemen s in ime se ies o ecas ing ha e hea ily elied on T ans o me -based
models, p ima ily due o hei powe ul a en ion mechanisms. Howe e , challenges such as
compu a ional ine iciency and he di icul y in cap u ing long- e m dependencies pe sis .
This sec ion del es in o a ious modelling s a egies designed o add ess hese challenges
and o he s, explo ing inno a ions in a en ion mechanisms, hie a chical s uc u es, and
hyb id app oaches – as shown in Figu e 5.3.
39
Figu e 5.3 – Di ision and Sub-di ision o Modelling Componen s, Showing he Numbe o
Models ha all unde each Sub-di ision
5.2.1. A en ion Mechanism
Time se ies o ecas ing has seen signi ican ad ancemen s wi h he in oduc ion o
T ans o me models. The a en ion mechanism, a co e componen o T ans o me s, has
been adap ed and e ol ed o be e handle he unique challenges p esen ed by ime se ies
da a, such as long- ange dependencies, empo al dynamics, and compu a ional e iciency.
5.2.1.1. Spa se Mechanisms
Spa se mechanisms ha e been inc easingly u ilized in ime se ies o ecas ing models o
imp o e e iciency and in e p e abili y by ocusing on he mos ele an pa s o he inpu
sequence. H. Zhou e al. (2020), in In o me , P obSpa se sel -a en ion uses a spa si y
measu emen based on he Kullback-Leible (KL) di e gence o iden i y he op-u mos
impo an que ies, p opo ional o he loga i hm o he sequence leng h. This mechanism is
adap ed in Y o me (Madhusudhanan e al., 2022) wi hin a U-Ne inspi ed mul i- esolu ion
amewo k, applied in bo h con ac ing (downscaling) and expanding (upscaling) pa hs, and
used in skip connec ions be ween co esponding encode and decode laye s, allowing he
model o le e age mul i- esolu ion ea u e maps, main aining consis ency in p edic ions.
Simila ly, X. Wang, Xia, & Deng (2023), in MSRN-In o me , employ he P obSpa se Sel -
a en ion om In o me , while FFT–In o me (J. Ma & Dan, 2023) combines i wi h a Hampel
il e o iden i y and il e s ou ou lie s, enhancing obus ness.
S. Wu e al. (2020), in AST, eplace he s anda d so max unc ion wi h α-en max,
in oducing spa si y in o he a en ion weigh s, assigning ze o weigh s o i ele an ime
40
s eps, hus imp o ing e iciency. Yang & Lu (2023), in Fo e o me , implemen explici spa se
mul i-head a en ion, e aining only he op a en ion sco es and no malizing hem, ensu ing
ocus on he mos ele an pa s o he sequence. Jawed & Schmid -Thieme (2022), in
GQFo me , employ spa se sel -a en ion o e a loga i hmically spaced subse o posi ions,
cap u ing long- ange dependencies and in oducing quan ile ansposed a en ion o
cap u e in e ac ions among di e en quan ile o ecas s.
N. Wang & Zhao (2023a), in En o me , in oduce Spa se Pe iodic Sel -A en ion, which
iden i ies key que ies wi h concen a ed a en ion dis ibu ions using KL di e gence,
cap u ing pe iodic con ex h ough au o-co ela ion and so max weigh ing. G. Tong e al.
(2024), in RSM o me , in oduce Residual Spa se A en ion (RSA) ha ocuses on compu ing
a en ion sco es o only he mos ele an que ies, using spa se ma ix ope a ions and
esidual connec ions o enhance e iciency and s abili y. W. Li, Meng, e al. (2023), in
Bid o me , in oduce bidi ec ional spa se sel -a en ion (Bis-A en ion) and he sha ed-QK
me hod o educe compu a ional complexi y and edundancy. Guo e al. (2021), in S acked-
In o me Ne wo k, al e s a en ion mechanisms by combining p obabilis ic spa se a en ion,
ha selec i ely compu es a en ion sco es o a subse o impo an okens, and sel -
a en ion dis illing, ha hal es he numbe o elemen s conside ed in successi e laye s,
signi ican ly imp o ing e iciency. M. Li e al. (2022), in Um o me , enhance e iciency and
accu acy wi h Sha ed Double-heads P obSpa se A en ion (SDHPA), combining p obabilis ic
spa si y and sha ed mul i-head a en ion ha uses wo a en ion heads o cap u e a b oade
ange o pa e ns while sha ing he same alues ac oss di e en heads, ensu ing ha
di e en heads a end o he same se o alues bu om di e en pe spec i es.
Addi ionally, g aph-based mechanisms inco po a e spa se a en ion o handle long sequence
ime-se ies o ecas ing e icien ly. Su e al. (2021), in AGCNT, use P obSpa se Adap i e
G aph Sel -A en ion, in eg a ing P obSpa se sel -a en ion o selec a spa se subse o key
elemen s, cons uc ing an adap i e g aph o ep esen dependencies, and applying g aph
con olu ion o agg ega e in o ma ion om neighbou ing nodes. Simila ly, Y. Wang e al.
(2024), in G aph o me , eplace adi ional sel -a en ion wi h g aph sel -a en ion, wo king
wi h he Tempo al Ine ia Module o cap u e spa ial dependencies and ecen da a ends,
ensu ing he model emains esponsi e o changes. C. Zhang e al. (2023), in Causal o me ,
employ P obSpa se Sel -a en ion in he Time Encode and adap s he Causal Encode 's
a en ion o c ea e a G ange causali y g aph, in eg a ing causal ela ionships.
5.2.1.2. Hyb id Mechanisms
Con olu ional mechanisms le e age con olu ional laye s o enhance he cap u e o local
dependencies and imp o e compu a ional e iciency. S. Li e al. (2019), in Log ans, eplace
adi ional do -p oduc sel -a en ion wi h spa se causal con olu ions o gene a e que ies
and keys, p ese ing empo al o de and cap u ing local con ex like shape. Shen & Wang
(2022), in TCCT, in oduce CSPA en ion, spli ing he inpu in o wo pa s: one p ocessed by
a ligh weigh 1×1 con olu ion o main ain o e all s uc u e, and he o he by adi ional sel -
41
a en ion o cap u e global dependencies. The ou pu s a e conca ena ed, educing
compu a ional and memo y cos s signi ican ly while imp o ing g adien low and cap u ing
bo h local and global dependencies.
Y. Li e al. (2022), in ACT, inco po a e a con olu ional a en ion block wi h a causal
con olu ional laye be o e sel -a en ion, e ec i ely cap u ing local dependencies and
add essing adi ional sel -a en ion's local con ex insensi i i y. Simila ly, X. Wang, Liu, Yang,
e al. (2023), in CN o me , employ s acked dila ed con olu ional blocks (SDCBs) o e ine
seasonal componen s and cap u e local and long- e m dependencies h ough con olu ions
in empo al and ea u e dimensions.
Z. Liu e al. (2024), in Hid o me , eplace adi ional mul i-head a en ion wi h a
combina ion o ecu ence and linea a en ion. Recu ence cap u es empo al
dependencies by p ocessing da a sequen ially, le e aging in o ma ion om p e ious ime
s eps, while linea a en ion educes compu a ional complexi y, scaling linea ly wi h
sequence leng h ins ead o quad a ically. H. Cao e al. (2023), in InPa o me , in oduce
In e ac i e Pa allel A en ion (InPa A en ion), ope a ing in bo h equency and ime
domains. I includes que y selec ion o educe compu a ional load, key- alue pai
comp ession h ough In e ac i e Pa i ioned Con olu ion (IPCon ), and a dual a en ion
s uc u e p ocessing ime-awa e and equency-awa e a en ion in pa allel.
5.2.1.3. F equency Mechanisms
H. Wu e al. (2021), in Au o o me , eplace adi ional sel -a en ion wi h an Au o-
Co ela ion mechanism, cap u ing pe iod-based dependencies by aligning and agg ega ing
simila sub-se ies om di e en pe iods, enhancing long- e m o ecas ing e iciency.
Simila ly, Z. Wang, Chen, Yang, e al. (2023), in HAFF, e ine his Au o-Co ela ion mechanism
o be e cap u e his o ical and nonlinea cha ac e is ics o elec ici y demand, ecognizing
long- ange dependencies and complex empo al pa e ns, and conside s ex e nal ac o s like
wo king days and seasonal changes o dynamically adjus o ecas s.
FED o me , in oduced by T. Zhou e al. (2022), uses F equency Enhanced A en ion (FEA)
blocks ha ope a e in he equency domain, ocusing on signi ican equency componen s
h ough Fou ie and wa ele ans o ms, selec i ely emphasizing impo an equencies
be o e ans o ming he da a back o he ime domain, using he In e se Fou ie T ans o m.
Simila ly, TD o me by X. Zhang e al. (2022) employs Fou ie a en ion o handle seasonal
componen s, ans o ming da a using he Fou ie T ans o m, compu ing a en ion sco es
among equency componen s, and no malizing hese sco es wi h so max o emphasize
signi ican pe iodic pa e ns be o e con e ing he da a back o he ime domain.
X. Wang, Liu, Du, e al. (2023), in CL o me , eplace s anda d global sel -a en ion wi h a
locally g ouped au oco ela ion mechanism wi hin equal-leng h segmen s, imp o ing he
cap u e o local empo al dynamics and mul i-scale dependencies. (Y. Cao & Zhao, 2023), in
DWT o me , inco po a e au o-co ela ion o iden i y and emphasize signi ican ime lags and
42
pe iodic pa e ns, olling he ime se ies o align simila phases and using So max-
no malized au o-co ela ion alues o ocus on key pe iods, enhancing long- e m
dependencies cap u e.
Ouyang e al. (2023a), in Rank o me , eplace Pea son co ela ion wi h Spea man’s Rank
Co ela ion (RankCo ) in sel -a en ion mechanisms, cap u ing nonlinea dependencies by
anking inpu da a and calcula ing au oco ela ion on hese anks using FFT and he Wiene -
Khinchin heo em, signi ican ly educing compu a ional complexi y and making he model
mo e obus o ou lie s and be e a iden i ying mono onic ela ionships.
5.2.1.4. Segmen -based Mechanisms
Y. Li, Lu, e al. (2023), in Con o me , in oduce a sliding-window a en ion mechanism ha
es ic s each poin o a end o a p ede ined numbe o nea by poin s, educing
compu a ional complexi y and e icien ly cap u ing ele an empo al dependencies o
long- e m o ecas ing. Simila ly, J. Chen e al. (2023), in Seg o me , in oduce SegA en ion,
which di ides he inpu sequence in o smalle segmen s and applies wo-dimensional
con olu ion ope a ions o he que y and key ma ices wi hin each segmen , cap u ing ine-
g ained empo al pa e ns mo e e ec i ely.
Ci s ea e al. (2022), in T i o me , in oduces a "Pa ch A en ion" mechanism, b eaking he
inpu ime se ies in o smalle segmen s (pa ches), each associa ed wi h a pseudo imes amp
ep esen ing he en i e pa ch. This mechanism p ocesses each pa ch by compu ing a en ion
sco es be ween he pseudo imes amp and he eal imes amps wi hin he pa ch, o ming a
iangula s uc u e whe e he inpu size o each subsequen laye dec eases, main aining
compu a ional e iciency while e ec i ely cap u ing ela ionships among di e en pa ches
and long- e m dependencies.
Shen e al. (2023), in FPP o me , in oduce Diagonal-Masked (DM) Sel -A en ion, combining
pa ch-wise and elemen -wise a en ion o cap u e local and global dependencies. This
mechanism masks diagonal elemen s in he a en ion sco e ma ix o p e en sel - ocus,
educing he impac o ou lie s. Y. Li, Qi, e al. (2023), in SMART o me , in oduce Window
A en ion (IWA), spli ing a en ion in o wo b anches: one o local sel -a en ion wi hin
non-o e lapping mul i-scale windows and ano he o global a en ion ac oss hese
windows, wi h he local b anch ocusing on de ailed dependencies wi hin windows and he
global b anch main aining b oade con ex ac oss he sequence.
Z. Wang e al. (2024), in PWD o me , in oduce he De o mable-Local (DL) Agg ega ion
Mechanism, which enhances he a en ion mechanism by adap i ely adjus ing he size o he
ime agg ega ion window, allowing he ans o me o ocus mo e p ecisely on ele an local
empo al pa e ns, imp o ing i s abili y o cap u e bo h sho - e m and long- e m
dependencies.
43
5.2.1.5. Hie a chical and Mul i-Scale Mechanisms
S. Liu e al. (2022), in Py a o me , in oduce he Py amidal A en ion Module (PAM),
o ganizing da a in o a py amidal g aph wi h laye s ep esen ing di e en esolu ions, om
ine o coa se. PAM is spa se and employs a dual a en ion mechanism ope a ing bo h in a-
scale, allowing nodes wi hin he same laye o ocus on local dependencies, and in e -scale,
connec ing nodes ac oss laye s o e icien da a in eg a ion. Simila ly, Du e al. (2023), in
P e o me , use Mul i-Scale Segmen -Co ela ion (MSSC) o calcula e segmen -wise
co ela ions, cap u ing local con inui y and educing complexi y, wi h he decode le e aging
P edic i e MSSC (P eMSSC).
Qu e al. (2024), in Fo wa d o me , in oduce Mul i-Scale Fo wa d Sel -A en ion (MSFSA),
cap u ing dependencies a mul iple scales by in eg a ing o wa d sliding window a en ion
o local dependencies by ocusing on a ixed numbe o immedia e u u e okens, o wa d
dila ed sliding window a en ion o medium- ange dependencies by a ending o non-
con iguous u u e okens, and o wa d global sliding window a en ion o long- ange
dependencies and signi ican bu in equen okens like holidays. Y. Zhang, Ma, e al. (2024),
in MTST, adap he a en ion mechanism o use ela i e posi ional encoding and pa ch
okeniza ion, pe o ming sel -a en ion on pa ch-le el okens in each b anch o cap u e
pe iodic and seasonal pa e ns ac oss mul iple esolu ions.
P. Chen e al. (2023), in Pa h o me , in oduce a dual a en ion sys em ope a ing on mul i-
scale di isions o ime se ies da a, using in a-pa ch a en ion o local dependencies wi hin
pa ches and in e -pa ch a en ion o global dependencies ac oss pa ches. Simila ly, Y.
Wang, Zhu, & Kang (2023), in DEST o me , in oduce Mul i-View A en ion (MVI-A en ion)
o cap u ing seasonal changes om mul iple pe spec i es in he equency domain and
Mul i-Scale A en ion (MSC-A en ion) o end componen s, using con olu ional il e s o
analyze sub- ends a a ious scales.
Zeng e al. (2022), in Mu o me , in oduce a mul i-g anula i y a en ion head mechanism
combined wi h mul iple pe cep ual domain (MPD) p ocessing. This app oach p ocesses inpu
da a in o di e en pe cep ual domains, which a e ed in o a ious a en ion heads, each
ocusing on dis inc in o ma ion. Addi ionally, Mu o me implemen s an a en ion head
p uning mechanism using he Kullback-Leible (KL) di e gence measu e o p une simila
edundan in o ma ion among he heads.
5.2.1.6. A en ion Mechanisms o Va iable In e ac ion
Lim e al. (2020), in TFT, in oduce In e p e able Mul i-Head A en ion ha enhances
in e p e abili y by sha ing alue ec o s ac oss all heads and using addi i e agg ega ion o
c ea e a uni ied a en ion ma ix. This app oach cla i ies which pa s o he inpu sequence
a e mos in luen ial, making empo al ela ionships and ea u e impo ance easie o
unde s and. Simila ly, Tang & Ma eson (2021), in P oT an, in eg a es s ochas ic la en
a iables in i s a en ion mechanism o model non-Ma ko ian dependencies, enabling he
50
exempli ies a hyb id app oach by employing a channel-independen se ing ini ially and la e
in eg a ing c oss-channel dependencies h ough a low- ank a en ion laye , combining he
s eng hs o bo h s a egies o obus and accu a e o ecas ing.
The ce ain y o e idence ega ding channel dependence and independence mechanisms is
subs an ia ed h ough a ious dedica ed abla ion s udies, ha p o ide obus e idence
suppo ing he design choices o hese models, con i ming he pi o al oles o channel
dependence and independence in op imizing ime se ies o ecas ing pe o mance.
5.2.4. Hie a chical and Mul i-Scale App oaches
As esea ch ad ances, se e al models ha e eme ged ha inco po a e hie a chical s uc u es
and mul i-scale me hods o imp o e p edic i e accu acy and e iciency. These app oaches
a y in implemen a ion, ocusing on di e en aspec s o empo al da a o handle complexi y
and scale e ec i ely.
5.2.4.1. Hie a chical Only
Se e al models combine hie a chical s uc u es wi h o he echniques o enhance hei
p edic i e capabili ies. H. Su e al. (2021), in AGCNT, use a hie a chical encode wi h s acked
P obspa se G aph Sel -A en ion Blocks (PGAB) o cap u e complex dependencies a
di e en abs ac ion le els, enhanced by con olu ion and pooling ope a ions, comp essing
and e ining he da a o e ec i e local and global pa e n ecogni ion. Simila ly, Zeng e al.
(2023), in Se o me , employ hie a chical posi ional encoding o p ocess long sequences by
segmen ing hem in o blocks, managing local dynamics wi hin each block (in ablock) and
ela ionships be ween blocks (in e block) o cap u e bo h local and global dependencies
e ec i ely, while main aining he in eg i y o posi ional in o ma ion ac oss di e en scales.
T. Li, Liu, e al. (2023), in MASTER, use in a-s ock and in e -s ock hie a chical agg ega ion o
model s ock co ela ions. I employs sel -a en ion o ine-g ained empo al de ails wi hin
indi idual s ocks and mul i-head a en ion o unde s and dynamic ela ionships be ween
mul iple s ocks, enhancing s ock p ice p edic ion accu acy.
Jawed & Schmid -Thieme (2022), in GQFo me , u ilize quan ile ansposed a en ion o
cap u e in e ac ions among di e en quan ile o ecas s, ocusing on pai wise in e ac ions
ac oss quan ile le els o accu a e modelling o p obabilis ic o ecas dis ibu ions. Simila ly,
Ye e al. (2023), in TDT, implemen a hie a chical me hod o o ecas ing wi h wo decode s:
one p edic ing he mean o cap u e cen al endency and egula pa e ns, and ano he
p edic ing s anda d de ia ion o quan i y unce ain y a ound he mean, p o iding s uc u ed
p obabilis ic o ecas s.
W. Li e al. (2023), in Bid o me , in eg a e hie a chical sel -a en ion dis illa ion,
downsampling sequences h ough max-pooling o ocus on dominan ea u es and educe
51
memo y usage p og essi ely. This app oach enhances scalabili y and pe o mance in long
sequence ime-se ies o ecas ing. J. Tong e al. (2023), in PDT ans, use a wo-laye app oach,
whe e T ans o me ex ac s empo al ea u es and makes p ima y o ecas s. Then a
condi ional gene a i e model e ines hese o ecas s using a ia ional in e ence. This allows
he model o handle complex dependencies and in e ac ions mo e e ec i ely
5.2.4.2. Mul i-Scale Only
O he models ocus solely on mul i-scale o mul i- esolu ion echniques o cap u e da a a
di e en le els o g anula i y. Shen & Wang (2022), in TCCT, u ilize he Pass h ough
mechanism o combine ou pu s om all encode sel -a en ion blocks, c ea ing a mul i-scale
ea u e map ha in eg a es bo h local de ails and b oade con ex ual in o ma ion. Simila ly,
Madhusudhanan e al. (2022), in Y o me , inspi ed by he U-Ne a chi ec u e, employ an
encode -decode s uc u e wi h con ac ing and expanding pa hs, downsampling inpu da a
o cap u e mul i-le el g anula i y and upscaling i o e ine p edic ions, handling bo h high-
le el ends and ine-g ained de ails.
Y. Li, Lu, e al. (2023), in Con o me , ep esen ime-se ies da a a a ious empo al
esolu ions (seconds, minu es, and hou s) by embedding mul i-scale ep esen a ions in o a
la en space. The model conca ena es and weigh s hese embeddings o cap u e di e se
empo al pa e ns ac oss di e en g anula i ies, c ucial o accu a e long- e m o ecas ing.
Sasal e al. (2022), in W-T ans o me s, u ilize Maximal O e lap Disc e e Wa ele T ans o m
(MODWT) o mul i- esolu ion analysis, decomposing ime se ies da a in o mul iple le els o
cap u e di e en equency componen s. This app oach iden i ies bo h high- equency
de ails and low- equency ends, enhancing he model's unde s anding o non-s a iona y
da a o mo e accu a e o ecas s.
X. Wang, Liu, Yang, e al. (2023), in CN o me , implemen s acked dila ed con olu ional
blocks (SDCBs) o expand he ecep i e ield and cap u e pa e ns a a ying scales. This
mul i-scale me hod iden i ies sho - e m luc ua ions and long- e m ends wi hin ime
se ies da a, in eg a ing in o ma ion ac oss empo al esolu ions o imp o e p edic ion
accu acy. X. Wang, Liu, Du, e al. (2023), in CL o me , in eg a e dila ed causal con olu ions
o expand he ecep i e ield exponen ially and cap u e empo al pa e ns a mul iple scales.
This app oach agg ega es in o ma ion o e di e en empo al esolu ions,. Addi ionally, he
locally g ouped au oco ela ion mechanism u he suppo s mul i-scale analysis by
segmen ing he ime se ies and calcula ing au oco ela ion wi hin hese segmen s
Qu e al. (2024), in Fo wa d o me , employ Mul i-Scale Fo wa d Sel -A en ion (MSFSA) wi h
o wa d sliding window a en ion o cap u ing local dependencies, o wa d dila ed sliding
window a en ion o medium- ange dependencies by in oducing gaps, and o wa d global
sliding window a en ion o long- ange dependencies. This mul i-scale app oach enables
e icien p ocessing o da a a di e en empo al esolu ions. H. Cao e al. (2023), in
InPa o me , use In e ac i e Pa i ioned Con olu ion (IPCon ) wi h mul iple pa allel
52
con olu ion b anches o cap u e ea u es a a ious esolu ions. This mul i- esolu ion
p ocessing ex ac s in o ma ion om ine de ails o b oade ends, enhancing he model's
abili y o p edic complex empo al pa e ns. Yu e al. (2023), in Ds o me , implemen
Double Sampling (DS) Blocks o mul i- esolu ion p ocessing, combining down-sampling o
cap u e global pa e ns and piecewise sampling o ine-g ained de ails. This app oach
ensu es e ec i e cap u e and u iliza ion o bo h b oad ends and de ailed a ia ions in ime
se ies da a o accu a e long- e m p edic ions.
Du e al. (2023), in P e o me , use he Mul i-Scale Segmen -Co ela ion (MSSC) mechanism
o segmen ime se ies in o exponen ially inc easing leng hs, cap u ing dependencies a
di e en empo al esolu ions. This mul i-scale segmen a ion enables he model o analyze
bo h ine-g ained and coa se-g ained pa e ns, enhancing o ecas ing by agg ega ing
ou pu s ac oss scales. Ouyang e al. (2023a), in Rank o me , use Mul i-Le el Decomposi ion
(MLDecomp) blocks, which employ mul iple mo ing a e age (MMA) il e s wi h di e en
ke nel sizes. This mul i-scale app oach decomposes ime se ies in o componen s such as
sho - e m seasonal a ia ions and long- e m ends, imp o ing o ecas ing accu acy by
unde s anding complex pa e ns in he da a.
Y. Wang, Zhu, & Kang (2023), in DEST o me , implemen Mul i-Scale A en ion (MSC-
A en ion) o cap u e sub- ends a a ious scales using con olu ional il e s wi h di e en
ecep i e ields. This app oach adap i ely models long- e m ends and dependencies a
mul iple esolu ions, p o iding a nuanced unde s anding o end pa e ns o imp o ed
o ecas ing. N. Wang & Zhao (2023a), in En o me , u ilize he Coa se Ma ching Module
(CMM) wi h con olu ional laye s o a ying ke nel sizes o ex ac ea u es a di e en
empo al esolu ions. This mul i-scale app oach cap u es dependencies a mul iple ime
scales, enhancing he model's abili y o cap u e sho - e m and long- e m pa e ns in ime-
se ies da a.
Lee e al. (2024), in TS-Fas o me , inco po a e he Sub Window Tokenize (SWT) and Pas
A en ion Decode o mul i- esolu ion p ocessing. The SWT segmen s inpu da a in o
smalle windows o cap u e pa e ns a di e en g anula i ies, imp o ing o ecas ing
accu acy o e a ying ime ho izons. Chen e al. (2023), in Pa h o me , di ide ime se ies
da a in o pa ches o a ious sizes o cap u e di e en empo al esolu ions simul aneously.
The mul i-scale ou e dynamically selec s he mos ele an pa ch sizes based on he inpu
da a’s cha ac e is ics, and he dual a en ion mechanism p ocesses hese pa ches o cap u e
bo h local and global dependencies.
5.2.4.3. Mul i-Scale and Hie a chical Techniques
Se e al models combine mul i-scale analysis wi h hie a chical echniques o cap u e complex
empo al pa e ns e ec i ely. H. Zhou e al. (2020), in In o me , use sel -a en ion dis illing,
whe e each laye condenses and abs ac s he in o ma ion om he p e ious laye , o
p og essi ely educe sequence leng h ia max-pooling, cap u ing mul i-scale in o ma ion
53
e icien ly. H. Wu e al. (2022), in Au o o me , u ilize mul i- esolu ion decomposi ion o
analyze ime se ies da a a di e en scales, enhancing p edic ion accu acy by ocusing on
a ious g anula i ies and in eg a ing pe iod-based dependencies.
Tang & Ma eson (2021), in P oT an, implemen a hie a chical s uc u e wi h laye s o
s ochas ic la en a iables, each laye cap u ing di e en le els o abs ac ion and empo al
pa e ns, imp o ing o ecas ing obus ness. S. Liu e al. (2022), in Py a o me , employ a
hie a chical, mul i- esolu ion app oach h ough Coa se -Scale Cons uc ion Module (CSCM),
ha c ea es a py amidal g aph s uc u e by downsampling he inpu da a o o m laye s ha
ep esen di e en esolu ions, om ine o coa se. Followed by PAM ha hen applies a
ailo ed a en ion mechanism bo h wi hin each laye (in a-scale) and ac oss di e en laye s
(in e -scale). Ci s ea e al. (2022), in T i o me , u ilize a iangula laye ing design, whe e he
inpu size o each subsequen laye is educed exponen ially by using only he pseudo
imes amps om he p e ious laye , o cap u e ea u es a mul iple empo al scales,
agg ega ing ou pu s om all laye s o comp ehensi e unde s anding.
T. Zhou e al. (2022), in FED o me , in eg a e Disc e e Wa ele T ans o m (DWT) o mul i-
esolu ion analysis and Mix u e O Expe Decomposi ion (MOEDecomp) o hie a chical
p ocessing, e ec i ely managing complex empo al in o ma ion o enhance o ecas ing
accu acy. S. Huang e al. (2022), in CEEMDAN, combine mul i- esolu ion decomposi ion
using CEEMDAN o decompose he ime se ies da a in o IMFs a di e en scales.
Subsequen ly, he hie a chical app oach is applied by analysing he complexi y o hese IMFs
wi h Sample En opy and hen selec i ely o ecas ing hem wi h ei he a T ans o me model
o high-complexi y componen s o a BPNN o low-complexi y ones, imp o ing accu acy
ac oss a ying da a complexi ies.
Shabani e al. (2023), in Scale o me , implemen a hie a chical, mul i-scale app oach by
i e a i ely e ining o ecas s a p og essi ely ine esolu ions, s a ing wi h a coa se scale. A
each scale, he inpu o he encode is downsampled, and he decode uses he upsampled
ou pu om he p e ious scale, ensu ing de ailed and e ined p edic ions. C oss-scale
no maliza ion is applied o main ain consis ency and educe e o s. Yang & Lu (2023), in
Fo e o me , use Mul i-Tempo al Resolu ion (MTR) Module ha ope a es by downsampling
he inpu sequence in o smalle subsequences a each le el o a bina y ee-like hie a chy.
Con olu ional laye s a e hen applied o hese subsequences o ex ac local empo al
ea u es a a ious scales. This hie a chical s uc u e allows he model o cap u e empo al
pa e ns a mul iple esolu ions. Zhong e al. (2023), in NT o me , deploy hie a chical
a en ion mechanisms o mul i- esolu ion/mul i-scale analysis, cap u ing sho - e m o
long- e m pa e ns simul aneously. This hie a chical app oach imp o es model obus ness
by o ganizing empo al dependencies e ec i ely.
Zeng e al. (2022), in Mu o me , a mul i-g anula i y a en ion head mechanism combined
wi h mul iple pe cep ual domain (MPD) p ocessing. The hie a chical amewo k o
Mu o me p unes edundan in o ma ion using he Kullback-Leible (KL) di e gence
54
measu e, ensu ing ha he a en ion heads cap u e unique and ele an ea u es ac oss
di e en scales, ensu ing de ailed pa e n ecogni ion. S. Ma e al. (2023), in TCLN, use a
mul i-ke nel CNN module o mul i-scale spa ial ea u e ex ac ion and hie a chical s acking
o encode laye s (LSTM and T ans o me ) o comp ehensi e empo al abs ac ion. This
combined app oach ensu es obus modelling o complex mul i a ia e ime se ies da a. Y. Li,
Qi, e al. (2023), in SMART o me , combine In eg a ed Window A en ion (IWA) o mul i-
scale dependency handling and a Semi-Au o eg essi e (SAR) Decode o hie a chical
sequence p ocessing. This in eg a ion enables e ec i e cap u e o bo h local and global
empo al pa e ns, enhancing o ecas ing accu acy.
Y. Cao & Zhao (2023), in DWT o me , u ilize wa ele decomposi ion, ha ans o ms he 1D
ime se ies in o 2D enso s ha cap u e di e en equency componen s, and Incep ion-like
con olu ional laye s wi hin he 2D-Fea u esBlock ha hen apply con olu ional il e s o
a ious sizes in pa allel, cap u ing ea u es a mul iple scales simul aneously. N. Wang &
Zhao (2023b), in Con olu ion T ans o me , in eg a e mul i- esolu ion echniques h ough a
Mul i-laye Fea u e Fusion componen , combining ea u e maps om di e en encode
laye s o cap u e de ailed local ea u es and global pa e ns. Hie a chical p ocessing in he
encode builds upon successi e laye s o comp ehensi e pa e n ex ac ion. B. Li, Cui, e al.
(2023), in Di Fo me , employ mul i- esolu ional di e encing ha inhe en ly c ea es a
hie a chical s uc u e by analysing he da a a di e en scales. Pa ch Me ging and Dynamic
Ranging u he suppo his hie a chy by segmen ing he ime se ies in o pa ches o a ying
sizes and adjus ing he a en ion spans dynamically based on he da a's pa e ns.
D. Cao e al. (2024), in TEMPO, implemen mul i-scale analysis by decomposing ime se ies
da a in o a ious esolu ions o cap u e di e en pa e ns and pe iodici ies. I s hie a chical
me hods o ganize da a in o le els o abs ac ion, enhancing o ecas ing accu acy by
in eg a ing b oad ends wi h speci ic pa e ns. Ni e al. (2024), in BasisFo me , in eg a e
adap i e sel -supe ised lea ning o cap u e basis ec o s ep esen ing mul i- esolu ion
empo al pa e ns. Hie a chical Coe Module selec s and weighs ele an pa e ns ac oss
di e en scales, enhancing o ecas ing accu acy h ough s uc u ed mul i-scale in eg a ion.
Shen e al. (2023), in FPP o me , uses hie a chical encode -decode s uc u es o p ocess
da a om ine o coa se esolu ions, cap u ing di e se empo al dynamics. This app oach
ensu es comp ehensi e ea u e in eg a ion ac oss mul iple scales, imp o ing o ecas ing
accu acy o complex ime-se ies da a.
G. Tong e al. (2024), in RSM o me , implemen a mul iscale o ecas ing s a egy, using up-
and-down sampling echniques, wi h hie a chical e inemen , ans o ming da a ac oss
esolu ions o cap u e empo al dependencies comp ehensi ely. This app oach ensu es
accu a e modelling o de ailed and b oad empo al in o ma ion in ime-se ies da a. Y. Zhang,
Wu, e al. (2024), in MTPNe , U ilizes a hie a chical py amid s uc u e o pa i ion ime se ies
da a in o pa ches a di e en scales, p ese ing spa ial and empo al dimensions. This mul i-
scale and hie a chical in eg a ion imp o es he model's abili y o lea n complex empo al
55
pa e ns and enhance o ecas ing accu acy. Y. Wang e al. (2024), in G aph o me , in eg a e
Mul i-Scale Fea u e Fusion o cap u ing empo al co ela ions a a ious g anula i ies and
hie a chical g aph sel -a en ion o p ocessing in e - a iable dependencies. This combined
app oach ensu es obus handling o empo al and spa ial dynamics in mul i a ia e ime
se ies o ecas ing. Z. Liu e al. (2024), in Hid o me , use a segmen -and-me ge a chi ec u e
o mul i- esolu ion/mul i-scale p ocessing, cap u ing local pa e ns a di e en empo al
scales. Hie a chical dual- owe a chi ec u e in eg a es in o ma ion om ime and equency
domains, enhancing accu acy and obus ness in long- e m ime se ies o ecas ing.
5.2.4.4. Discussion
The end in ime-se ies o ecas ing models shows a clea shi owa ds inco po a ing
hie a chical and mul i-scale echniques o handle he complexi y and scale o empo al da a.
These models use hie a chical s uc u es o manage dependencies a a ious abs ac ion
le els, enhancing he abili y o cap u e in ica e pa e ns and in e ac ions wi hin he da a.
Mul i-scale app oaches a e equally p e alen , enabling models o analyze da a a di e en
g anula i ies, om ine de ails o b oad ends. By combining hese me hods, esea che s
can le e age he s eng hs o bo h hie a chical o ganiza ion and mul i- esolu ion analysis,
leading o mo e comp ehensi e and eliable o ecas ing models.
Rega ding he ce ain y o e idence, only he ans o me s wi h dedica ed componen s o
mul i-scale echniques we e conside ed, as he ones ha do i indi ec ly, o example
h ough embedding, ha e hei dedica ed s udies ega ding ce ain y o e idence. he
abla ion s udies o ans o me s ha di ec ly inco po a e mul i-scale componen s
demons a e hei signi ican impac on ime-se ies o ecas ing accu acy. Mu o me 's mul i-
g anula i y a en ion mechanism, based on Mul iple Pe cep ual Domains (MPD), imp o es
MSE and MAE by educing edundancy in a en ion heads. Fo wa d o me 's Mul i-Scale
Fo wa d Sel -A en ion (MSFSA) mechanism cap u es sho - e m and long- e m
dependencies e ec i ely h ough op imized pa ame e uning. P e o me 's mul i-scale
s uc u e ex ac s dependencies a a ious empo al esolu ions, leading o be e
pe o mance on complex da ase s. Scale o me combines a mul i-scale amewo k wi h
adap i e loss o enhanced esul s. Fo e o me 's Mul i-Tempo al Resolu ion (MTR) module is
c ucial o cap u ing de ailed empo al ea u es, signi ican ly imp o ing o ecas ing accu acy.
RSM o me ’s mul iscale o ecas ing and esidual spa se a en ion e ine p edic ions
i e a i ely, ocusing on ele an in o ma ion. Las ly, MTPNe 's mul i-scale ans o me
py amid ou pe o ms single-scale models, highligh ing he impo ance o mul i-scale
empo al dependency lea ning. These s udies con i m he alue o di ec mul i-scale
app oaches in de eloping obus and p ecise ime-se ies o ecas ing models.
56
5.2.5. Hyb idiza ion
In he ealm o ime se ies o ecas ing, hyb idiza ion s ands as a pi o al s a egy, combining
he s eng hs o ans o me wi h he s eng hs o di e en models.
5.2.5.1. Posi ional Encodings
As co e ed he sec ion 5.1.1 Posi ional Embedding, a ious hyb id app oaches ha e been
employed o enhance ime se ies o ecas ing. Fo ins ance, Y. Cao & Zhao (2023), in
DWT o me , use 2D-Fea u esBlock wi h mul i-scale con olu ional il e s o cap u e complex
pa e ns in ime se ies da a. Simila ly, X. Wang, Liu, Du, e al. (2023), in CL o me , apply
dila ed causal con olu ions o cap u e mul i-scale empo al pa e ns, e ining inpu da a by
emphasizing egula and p edic able pa e ns.
Y. Zhang, Wu, e al. (2024), in MTPNe , ake a di e en app oach by using a 1-laye CNN o
p ojec ime se ies da a in o a highe -dimensional space, aiding in cap u ing ine o coa se
empo al dependencies. Building on his idea, Domínguez-Cid e al. (2023), in TEFNEN,
employ ully connec ed single-laye neu al ne wo ks o embedding and an RNN laye o
cap u ing sequen ial and empo al dependencies.
Addi ionally, Cen & Lim (2024), in Pa chTCN-TST, use TCN wi h causal con olu ions o embed
empo al in o ma ion, p ese ing empo al o de and ocusing on local dependencies.
Finally, (J. Zhu e al., 2023), in SL-T ans o me , in eg a e ans o me s wi h LSTM ne wo ks
o ea u e ex ac ion and empo al sequence p ocessing, enhancing powe o ecas ing
accu acy.
5.2.5.2. A en ion Mechanisms
As co e ed in he sec ion 5.2.1.2 Hyb id Mechanisms, a ious app oaches ha e been
adop ed o enhance he a en ion mechanisms. Shen & Wang (2022), in TCCT, inco po a e a
ligh weigh 1×1 con olu ion o main ain s uc u e alongside sel -a en ion o global
dependencies, educing compu a ional cos s. Simila ly, Y. Li e al. (2022), in ACT, u ilize a
con olu ional a en ion block wi h a causal con olu ional laye o cap u e local con ex
be o e sel -a en ion.
Ano he app oach is seen in Z. Liu e al. (2024), in Hid o me , use ecu ence o cap u es
empo al dependencies by p ocessing da a sequen ially, le e aging in o ma ion om
p e ious ime s eps, while linea a en ion educes compu a ional complexi y, scaling
linea ly wi h sequence leng h. H. Cao e al. (2023), in InPa o me , u ilize In e ac i e
Pa i ioned Con olu ion (IPCon ) o key- alue pai comp ession in i s dual a en ion
s uc u e.
Mo eo e , no explici ly co e ed in ha sec ion, Dai e al. (2023), in PDF, combine sel -
a en ion wi h con olu ional laye s in he Dual Va ia ions Modeling Block o p ocess 2D
enso s and cap u e local pa e ns. Expanding on con olu ional in eg a ion, (B. Li, Cui, e al.,
57
2023), in Di o me , in eg a e CNN elemen s ia Δ-Lagging o cap u e local pa e ns and
adjus ocus dynamically in ime se ies.
Fu he enhancing long- e m dependency cap u e, N. Wang & Zhao (2023b), in Con olu ion
T ans o me , use causal dila ed con olu ions be ween sel -a en ion modules o enhance
long- e m dependency cap u e. In con as , Ci s ea e al. (2022), in T i o me , employ
ecu ency because Pa ch A en ion makes i ha de o cap u e ela ionships among
di e en pa ches and long- e m dependencies. I connec s pseudo imes amps o
consecu i e pa ches o main ain empo al in o ma ion low, using a ga ing mechanism o
con ol he low o in o ma ion om one pseudo imes amp o he nex . Las ly, (H. Su e al.,
2021), in AGCNT, employ 1-D con olu ional laye s o p ocess ou pu s om adap i e g aph
sel -a en ion laye s.
5.2.5.3. Mul i-Scale App oaches
Va ious models le e age mul i-scale and mul i- esolu ion echniques o enhance hei
pe o mance, as co e ed In sec ion 5.2.4 Hie a chical and Mul i-Scale App oaches. N. Wang
& Zhao (2023a), in En o me , u ilize he Coa se Ma ching Module wi h con olu ional laye s
o a ying ke nel sizes o cap u e dependencies a mul iple ime scales. Simila ly, H. Cao e
al. (2023), in InPa o me , use In e ac i e Pa i ioned Con olu ion wi h mul iple pa allel
con olu ion b anches o mul i- esolu ion p ocessing o empo al pa e ns.
Building on hese app oaches, Y. Wang, Zhu, & Kang (2023), in DEST o me , implemen
Mul i-Scale A en ion wi h con olu ional il e s o di e en ecep i e ields o adap i ely
model ends a a ious scales. Likewise, Yang & Lu (2023), in Fo e o me , use a Mul i-
Tempo al Resolu ion Module wi h con olu ional laye s o ex ac ea u es om
downsampled subsequences in a bina y ee-like hie a chy.
Addi ionally, X. Wang, Xia, & Deng (2023), in MSRN-In o me , use 1D CNNs wi h mul i-scale
con olu ional ke nels o cap u e ea u es a a ious scales and local dependencies. Fu he
enhancing mul i- esolu ion p ocessing, N. Wang & Zhao (2023b), in Con olu ion
T ans o me , in eg a e mul i- esolu ion echniques h ough Mul i-laye Fea u e Fusion,
combining ea u e maps om di e en encode laye s o de ailed local and global pa e n
cap u e. Las ly, S. Ma e al. (2023), in TCLN, use a mul i-ke nel CNN module o mul i-scale
spa ial ea u e ex ac ion and hie a chical s acking o encode laye s o comp ehensi e
empo al abs ac ion.
5.2.5.4. Decomposi ion
In he ealm o decomposi ion o ime se ies o ecas ing, di e en app oaches ha e been
u ilized o b eak down da a in o i s undamen al componen s. J. Tong e al. (2023), in
PDT ans, use con olu ion ope a ions and MLP o decompose ime se ies in o end and
seasonali y componen s, espec i ely. Simila ly, X. Wang, Liu, Yang, e al. (2023), in
58
CN o me , employ s acked dila ed con olu ional blocks o e ine seasonal componen s and
cap u e bo h local and long- e m dependencies.
5.2.5.5. Fea u e Focus
Fea u e selec ion and ex ac ion play c ucial oles in imp o ing he accu acy and e iciency.
Lim e al. (2020), in TFT, pe o m ea u e selec ion h ough a iable selec ion ne wo ks using
Ga ed Residual Ne wo ks (GRNs) o dynamically de e mine he ele ance o inpu ea u es.
Building on his, M. Li e al. (2022), in Um o me , employ a Fea u e Selec ion Ne wo k wi h
GRNs and GLUV3 o op imize ea u e selec ion and manage nonlinea ela ionships wi h
ad anced ga ing mechanisms.
A simila app oach is aken by S. Zhu e al. (2023), in MR- ans o me , which u ilize empo al
con olu ion and a iable a en ion mechanisms o cap u e bo h consis en and speci ic
a iable ea u es. Enhancing empo al ea u e ex ac ion u he , (Fan e al., 2023), in
TED o me , enhance empo al ea u e ex ac ion using Tempo al Con olu ional Ne wo ks
(TCN) o cap u e local pa e ns and ends. The TCN p ocesses end componen s wi hin he
encode o e ec i ely cap u e long- ange dependencies and local ea u es, in eg a ing TCN
in o bo h he encode and decode .
5.2.5.6. Modeling
In he modelling o ime se ies o ecas ing, se e al inno a i e hyb id app oaches ha e been
de eloped. Fo ins ance, Zeng e al. (2023), in Se o me , use ecu si e in e block posi ion
encoding o manage ela ionships be ween da a blocks, main aining global sequence con ex
and long- ange dependencies. Simila ly, S. Huang e al. (2022), in CEEMDAN, combine Back
P opaga ion Neu al Ne wo k (BPNN) o o ecas ing low-complexi y subsequences wi h
ans o me s o high-complexi y segmen s.
Taking a di e en app oach, X. Zhang e al. (2022), in TD o me , u ilize a Mul i-Laye
Pe cep on (MLP) o handle he end componen o ime-se ies da a. The ime se ies is
decomposed in o end and seasonal componen s, wi h he MLP p edic ing u u e alues by
lea ning complex, non-linea ela ionships in he end da a. Building on he concep o
decomposi ion, Y. Li, Lu, e al. (2023), in Con o me , use RNNs and GRUs wi hin he
S a iona y and Ins an Recu en Ne wo k (SIRN) o cap u e bo h long- e m and sho - e m
empo al pa e ns. The end componen is p ocessed by a GRU o e ine long- e m
pa e ns, while he seasonal componen unde goes u he decomposi ion h ough
con olu ion and GRU blocks o cap u e sho - e m a ia ions.
Mo eo e , Z. Zhang e al. (2024), in SageFo me , in eg a e G aph Neu al Ne wo ks (GNNs)
wi h he adi ional ans o me encode o cap u e in e -se ies dependencies in
mul i a ia e ime se ies da a. I uses global okens and i e a i e message passing o acili a e
in e -se ies in e ac ions, wi h GNNs handling hese in e ac ions and he ans o me
59
managing in a-se ies in e ac ions, ul ima ely simpli ying he decoding p ocess wi h a linea
Fo ecas ingHead.
Enhancing deep in o ma ion ex ac ion, Jiang e al. (2022), in Deep Au o o me , enhance
he basic Au o o me amewo k by s a egically in eg a ing Mul i-Laye Pe cep ons (MLPs)
o imp o e deep in o ma ion ex ac ion. This in eg a ion enhances ea u e ex ac ion
e iciency, con ibu ing o supe io pe o mance in Ve y Sho -Te m Load Fo ecas ing
(VSTLF) and Sho -Te m Load Fo ecas ing (STLF).
5.2.5.7. Hyb idiza ion wi h S a is ical and Ma hema ical Models
(Tang & Ma eson, 2021), in P oT an, in eg a e ans o me s wi h s a e-space models
(SSMs), combining a en ion mechanisms wi h he s uc u ed app oach o SSMs o cap u e
long- e m dependencies and manage unce ain ies unce ain ies h ough la en a iables.
Simila ly, (Y. Lin e al., 2021), in SSDNe , me ge ans o me s wi h SSMs, whe e he
ans o me ex ac s empo al pa e ns and gene a es la en ep esen a ions, which a e
hen ed in o he SSM o end, seasonali y, and esidual decomposi ion. This in eg a ion
le e ages bo h complex ea u e ex ac ion and s uc u ed, in e p e able decomposi ion o
accu a e o ecas s.
Taking a di e en app oach, H. Wu e al. (2023), in D-T ans o me , in eg a e he Du ing
equa ion in o he ans o me 's ini ializa ion, using his nonlinea di e en ial equa ion o se
ini ial weigh s o he encode , decode , and ully connec ed laye s. This app oach cap u es
complex dynamical beha iou s, enhancing he model’s abili y o handle nonlinea i ies in
ime se ies da a, leading o as e aining con e gence and imp o ed p edic ion accu acy.
Mo eo e , C. Zhang e al. (2023), in Causal o me , combine empo al and causal ea u es
using a Time Encode wi h P obSpa se Sel -a en ion o empo al dependencies and a
Causal Encode wi h a G ange causali y g aph o iden i y and in eg a e causal ela ionships.
5.2.5.8. Discussion
Hyb idiza ion s a egies in ans o me models o ime se ies o ecas ing enhance
pe o mance by in eg a ing a ious neu al ne wo k a chi ec u es, s a is ical models, and
ma hema ical equa ions. Combining con olu ional laye s wi h ans o me s, as seen in
DWT o me and CL o me , cap u es bo h local de ails and b oade empo al dependencies.
Simila ly, in eg a ing RNNs and GRUs in models like TEFNEN and Con o me imp o es he
cap u e o sequen ial dependencies and long- e m pa e ns, essen ial o accu a e
o ecas ing.
Addi ionally, hyb idiza ion wi h s a is ical and ma hema ical models adds s uc u e and
in e p e abili y. Models such as P oT an and SSDNe combine ans o me s wi h s a e-space
models (SSMs) o ad anced ea u e ex ac ion and s uc u ed decomposi ion, leading o
mo e accu a e o ecas s. The inclusion o he Du ing equa ion in D-T ans o me highligh s
66
Ashok e al. (2024), in TACTiS-2, apply cu iculum lea ning h ough a wo-s age aining
p ocess o simpli y and imp o e he lea ning o complex dependencies. In he i s s age, he
model ains he pa ame e s o he ma ginal dis ibu ions independen ly, igno ing he
dependencies be ween a iables. Once he ma ginals a e accu a ely lea ned, he second
s age ocuses on aining he copula pa ame e s, which cap u e he dependencies, while
keeping he ma ginal pa ame e s ixed. This sequen ial app oach helps he model con e ge
as e and mo e e ec i ely by b eaking down he complex lea ning ask in o manageable
s eps.
5.4.2. Gene a i e T aining
Gene a i e aining has gained a en ion as i allows he model o p edic ou pu s in a single
o wa d pass, educing in e ence ime and mi iga ing e o accumula ion.
Some models in eg a e a Gene a i e Ad e sa ial Ne wo k (GAN) amewo k wi h he
T ans o me . S. Wu e al. (2020), in AST, and Y. Li e al. (2022), in ACT, implemen his by
ha ing he gene a o o ecas s u u e alues, and he disc imina o di e en ia e hese om
ac ual da a, enhancing he model's eplica ion o he da a dis ibu ion.
O he models use a Gene a i e Decode . (H. Zhou e al., 2020), in In o me , and Guo e al.
(2021), in S acked-In o me Ne wo k eed he decode wi h a combina ion o s a okens
and placeholde s o he a ge sequence, wi h he la e using a en ion mechanisms o
ocus on ele an pa s o he inpu sequence. W. Li e al. (2023), in Bid o me , use
gene a i e in e ence, di iding he inpu sequence in o a s a oken sequence and a
placeholde o he a ge sequence, p ocessed using masked mul i-head a en ion.
Su e al. (2021), in AGCNT, implemen gene a i e aining h ough i s gene a i e in e ence
ea u e in he decode , allowing he gene a ion o u u e alues in a single ope a ion.
Simila ly, C. Zhang e al. (2023), in Causal o me , employ a gene a i e app oach in i s
decode , whe e each head ocuses on a speci ic a iable, conside ing i s causal in luences,
and p oduces an ex ended sequence in pa allel, enhancing in e ence speed and accu acy by
in eg a ing causal insigh s and applying spa si y egula iza ion. Yu e al. (2023), in Ds o me ,
employ a gene a i e decode based on MLP o ans o m p ocessed ea u e ec o s in o inal
p edic ion ou pu s. This MLP-based decode akes he comp ehensi e ep esen a ions
gene a ed by he Tempo al Va iable A en ion (TVA) blocks, which in eg a e bo h empo al
and a iable in o ma ion, and syn hesizes hem in o p edic ed alues.
And inally, o he models decide o inco po a e La en Va iable Models. J. Tong e al. (2023),
in PDT ans, in eg a e a condi ional gene a i e model wi h a T ans o me using a a ia ional
au oencode (VAE) amewo k o e ine ini ial o ecas s by modelling hei p edic i e
dis ibu ion h ough a ia ional in e ence, p oducing p obabilis ic o ecas s. Y. Li, Lu, e al.
(2023), in Con o me , inco po a e gene a i e elemen s h ough a no malizing low block,
modelling he dis ibu ion o u u e se ies di ec ly om la en s a es, cap u ing unde lying
pa e ns and unce ain ies, enhancing obus ness in long- e m o ecas ing.
67
5.4.3. P obabilis ic Fo ecas ing
P obabilis ic o ecas ing p o ides a comp ehensi e unde s anding o u u e unce ain ies,
enabling mo e in o med decision-making and isk assessmen . Tang & Ma eson (2021), in
P oT an, use s ochas ic la en a iables and a ia ional in e ence o cap u e unce ain ies in
ime se ies da a, gene a ing di e se long- e m o ecas s and p edic ion in e als o
enhanced in e p e abili y. Jawed & Schmid -Thieme (2022), in GQFo me , gene a e mul iple
quan ile es ima es ac oss he o ecas ho izon using implici quan ile ne wo ks and a mul i-
ask loss unc ion wi h Con inuous Ranked P obabili y Sco e (CRPS) o es ima e u u e
alues' condi ional dis ibu ion.
Lin e al. (2021), in SSDNe , le e age he T ans o me 's abili y o ex ac empo al pa e ns
and he S a e Space Model's (SSM) p obabilis ic es ima es. The T ans o me p ocesses
his o ical da a and co a ia es o gene a e la en ep esen a ions, which a e hen used o
es ima e he SSM pa ame e s. The SSM models he ime se ies as a combina ion end,
seasonali y, and Gaussian-dis ibu ed esiduals. This enables SSDNe o p edic u u e alues
and hei dis ibu ion, cap u ing unce ain y using a combined loss unc ion o MAE and
Nega i e Log-Likelihood (NLL). Ye e al. (2023), in TDT, p edic he p obabili y dis ibu ion o
u u e alues by sequen ially o ecas ing mean and s anda d de ia ion, e ec i ely
quan i ying unce ain y wi h con idence in e als and isk assessmen s.
5.4.4. Discussion
The sec ion on Types o Lea ning showcases how a ious inno a i e s a egies a e ailo ed o
mee di e en needs. Fede a ed lea ning enhances p i acy by decen alizing da a
p ocessing, ans e lea ning makes use o p e-exis ing knowledge, mul i- ask lea ning
imp o es gene aliza ion, sel -supe ised lea ning de i es obus da a ep esen a ions om
unlabelled da ase s, and end- o-end and cu iculum lea ning handles complex dependencies
and ensu e consis en aining.
Gene a i e aining, inco po a ing echniques like GANs and gene a i e decode s, is gaining
a en ion o educing in e ence ime and mi iga ing e o accumula ion, while p obabilis ic
o ecas ing me hods emphasize cap u ing unce ain ies h ough ad anced mechanisms like
s ochas ic la en a iables and s a e-space models, p o iding comp ehensi e insigh s in o
u u e p edic ions.
Rega ding he ce ain y o e idence, he ypes o lea ning a e di icul o es hei impac
since o es he p obabilis ic o ecas ing he abla ion would lead o a di e en me ic and
s anda d supe ised lea ning canno easily eplace Inno a i e Lea ning S a egies. As such
mos o hem lack dedica ed abla ion s udies excep o AST, ACT and In o me ha show
ha he gene a i e aining alle ia es he e o accumula ion, as well as TEMPO and TACTiS-
2 ha show ha he exclusion o he p omp componen in abla ion s udies led o a
de e io a ion in p edic i e accu acy and ha he wo-s age cu iculum signi ican ly
con ibu es o he model's pe o mance and e iciency, espec i ely. O e all, while dedica ed
68
abla ion s udies a e sca ce, he a ailable e idence o hese speci ic models highligh s he
bene i s o hei unique componen s and he good pe o mance o he models also implici ly
does he same.
5.5. REPEATED TRANSFORMERS
As p e iously discussed, mos esea ch in his a ea ocuses on c ea ing ans o me s ailo ed
o speci ic pu poses, a he han imp o ing exis ing ans o me s. Consequen ly, he e a e
ela i ely ew s udies ha apply exis ing ans o me s o o he domains, hus expanding he
knowledge abou hei pe o mance.
The Tempo al Fusion T ans o me (TFT) has been widely applied o a ious sec o s,
showcasing i s u ili y in mul iple con ex s. Fo ins ance, i has been e ec i ely u ilized o
o ecas ing Bi coin p ice ends (Amadeo e al., 2023), weekly a ia ions in hospi al pa ien
coun s (Caldas & Soa es, 2023), and shi s in ene gy demands wi hin powe dis ibu ion
ne wo ks (Liao & Radhak ishnan, 2022). Simila ly, i has p o en aluable in heal hca e o
p edic ing i al sign ajec o ies (Phe i ikun e al., 2021) and in ag icul u e o o ecas ing
whea yields (Junanka e al., 2023). Addi ionally, i s applica ion ex ends o p edic ing he
s a e o cha ge o sodium-sulphu (NaS) ba e ies (Alma zooqi e al., 2023) and ene gy
consump ion pa e ns (Ji anon e al., 2023), as well as wind powe o ecas ing in Sou h
A ica ( an Hee den e al., 2023). This di e se applica ion spec um illus a es he TFT's
capaci y o adap o di e en da a cha ac e is ics and o ecas ing equi emen s.
The In o me ans o me , meanwhile, was used in mo e echnical applica ions such as
p edic ing mo o bea ing ib a ions (Z. Yang e al., 2022) and hea ing loads (M. Gong e al.,
2022). No ably, i s use in hea ing load o ecas ing was enhanced by he in oduc ion o a
Rela i e Posi ion Encoding algo i hm, highligh ing a specialized adap a ion.
Las ly, he FED o me was speci ically employed o p edic wind speed (Deng e al., 2022),
showing i s po en ial in en i onmen al and wea he - ela ed o ecas ing.
All hese show he lexibili y and po en ial o ans o me o be applied o di e se a eas o
eal wo ld da ase s.
Also, mos o he ans o me s use he Vanilla T ans o me as hei basis, only a ew
ans o me s apply hei inno a ions o o he exis ing ans o me s. This is ep esen ed in
Figu e 5.5, whe e all he ans o me s based on di e en ans o me s and wha ans o me
hey a e based on a e shown. The a ow o GPT is do ed o d aw a en ion ha GPT was no
c ea ed o ime se ies o ecas ing, unlike he o he models.
69
Figu e 5.5 – Diag am o he Inspi a ion o Di e en T ans o me s
70
6. CONCLUSION AND FUTURE WORK
Recen ad ancemen s in ans o me models o ime se ies o ecas ing showcase signi ican
ends, inno a ions, and me hodological ad ancemen s. Enhanced inpu ep esen a ions,
such as adap i e and lea nable embeddings and mul i- esolu ion embeddings, imp o e
model adap abili y and cap u e empo al dependencies mo e e ec i ely. Techniques like
segmen -based app oaches and decomposi ion me hods isola e ends and pa e ns,
enhancing accu acy and obus ness. Inno a i e a en ion mechanisms, including spa se and
hyb id a en ion, add ess compu a ional e iciency and cap u e bo h local and global
dependencies. Hie a chical and mul i-scale app oaches decompose da a in o mul iple
esolu ions, managing dependencies a a ious le els and imp o ing p edic i e accu acy.
E ec i e ea u e selec ion and co a ia e in eg a ion dynamically de e mine ele an inpu s,
educing noise and enhancing model pe o mance.
Addi ionally, ad ancemen s in handling channel dependence and independence ha e been
made. Some models emphasize cap u ing in e - a iable ela ionships o imp o ed
o ecas ing accu acy, while o he s ocus on p e en ing c oss-channel in e e ence o
enhance obus ness. Hyb idiza ion s a egies in eg a e ans o me s wi h o he models like
CNNs and RNNs o comp ehensi e ea u e ex ac ion and empo al abs ac ion, le e aging
he s eng hs o di e en a chi ec u es o imp o ed pe o mance. Regula iza ion
echniques, no maliza ion me hods, and ad anced op imiza ion s a egies s abilize aining
and p e en o e i ing. Di e se loss unc ions balance accu acy and obus ness. Inno a i e
lea ning s a egies, such as ans e lea ning, enhance gene aliza ion and e iciency, while
p obabilis ic o ecas ing elemen s allow use s o us model decisions and cap u e o ecas
unce ain y. Gene a i e aining enhance obus ness. Mo eo e , he e is a s ong emphasis
on educing compu a ional complexi y om quad a ic o linea , as shown in Appendix A
(Complexi y Equa ions), and op imizing model s uc u es o lowe compu a ional cos s
while main aining pe o mance.
Howe e , he e is no single s a e-o - he-a ans o me model o ime se ies o ecas ing
due o ac o s like s ochas ici y, a ying da a cha ac e is ics, he use o unique p i a e
da ase s and di e en e alua ion me ics. Selec ing a ans o me model should depend on
he speci ic needs and da a cha ac e is ics. Fo p i acy conce ns, he Fede a ed T ans o me
is sui able. Fo eal- ime o ecas ing using inc emen al aining, InT ans is he only op ion o
now. Fo p obabilis ic o ecas ing, GQFo me and TDT a e wo h conside ing. Fo non-
s a iona y da a, W- ans o me s o he non-s a iona y ans o me a e good op ions. Fo
non-linea da a, DDP o me is e ec i e. I you need o use co a ia es, TFT is a s ong choice.
Fo da ase s wi h high in e - a iable co ela ion, JTFT and CN o me a e excellen . Based
solely on he lowes MSE me ics on he 9 mos common da ase s, MTST and Hid o me a e
op choices o mul i a ia e o ecas ing, while InPa o me excels o uni a ia e o ecas ing.
71
Despi e hese ad ancemen s, challenges and gaps emain. The e is limi ed unde s anding o
ans o me models' pe o mance ac oss di e se domains and da ase s, as mos s udies
ocus on ailo ed models a he han c oss-domain e alua ions. The lack o s anda dized
benchma ks complica es model assessmen , and he e is a signi ican need o mo e
anspa en and explainable models. While some models ha e a emp ed o imp o e
in e p e abili y h ough specialized a en ion mechanisms and causal in e ence, he e is s ill
a need o mo e anspa en and explainable models.
Addi ionally, he bias analysis e eals anspa ency issues as a conside able amoun o
pape s does no disclosing he sou ces o unding and a possible egional bias, as he e is
p edominance o p i a e da ase s o m only one coun y. The pape s also o en only show
he bes esul s ob ained, wi h some pape s no making he pa ame e s used public. This
highligh s he need o mo e igo ous and anspa en e alua ion p ac ices o ensu e obus
and unbiased model assessmen s.
Hie a chical and mul i-scale app oaches need u he explo a ion o op imize con igu a ions
and ade-o s be ween complexi y and pe o mance. Scalabili y and compu a ional
e iciency a e majo conce ns; despi e he in oduc ion o spa se and hyb id a en ion
mechanisms, la ge-scale and eal- ime implemen a ions emain challenging. Handling
missing da a and obus ness o noisy da ase s a e c i ical a eas needing a en ion, as mos
models assume well-s uc u ed, clean da a, which is un ealis ic in eal-wo ld scena ios.
F om he u u e esea ch di ec ions s a ed by he pape s he e we e some ends. A
signi ican ocus is on explo ing be e spa si y s a egies in sel -a en ion mechanisms and
adap ing models o handle small da ase s e ec i ely. This includes de eloping da a-e icien
me hods, le e aging inc emen al compu a ions, and adap i ely lea ning hype -pa ame e s
om he da a o imp o e pe o mance ac oss di e en applica ions. Enhancing he
in e p e abili y o models and s udying hei lea ning dynamics is ano he common heme,
wi h e o s di ec ed owa ds imp o ing he unde s anding o model decisions, p e en ing
o e -s a iona iza ion, and op imizing mul i-headed a en ion mechanisms.
Also, some ans o me s o ime se ies o ecas ing we e inspi ed by exis ing s a egies used
by ans o me s in o he domains. Now, he e a e s a e-o - he-a ans o me s speci ically
designed o ime se ies o ecas ing ha ha e no ye been es ed in o he ields, such as
compu e ision (CV) and na u al language p ocessing (NLP). Con e sely, he e a e new
s a e-o he a models in di e en a eas ha s ill ha en’ been es ed o ime se ies
o ecas ing.
Fo u u e wo k speci ic o his pape , i would be in e es ing o b oaden i s scope o
ans o me s o a ious a eas. Examining how hese models adap o di e en domains
could unco e inno a ions o app oached wi h po en ial o ans e o o he a eas.
72
BIBLIOGRAPHICAL REFERENCES
Abadi, A., Rajabioun, T., & Ioannou, P. A. (2015). T a ic Flow P edic ion o Road
T anspo a ion Ne wo ks Wi h Limi ed T a ic Da a. IEEE T ansac ions on In elligen
T anspo a ion Sys ems, 16(2), 653–662. IEEE T ansac ions on In elligen T anspo a ion
Sys ems. h ps://doi.o g/10.1109/TITS.2014.2337238
Alma zooqi, A. H., Alhusin, M. O., Nikolakakos, I. P., Husnain, A., & Albesh , H. M. (2023).
Imp o ed NaS Ba e y S a e o Cha ge Es ima ion by Means o Tempo al Fusion T ans o me .
2023 IEEE Texas Powe and Ene gy Con e ence (TPEC), 1–6.
h ps://doi.o g/10.1109/TPEC56611.2023.10078625
Amadeo, A. J., Sien o, J. G., Eikwine, T. A., Diana, & Pa monangan, I. H. (2023). Tempo al
Fusion T ans o me o Mul i Ho izon Bi coin P ice Fo ecas ing. 2023 IEEE 9 h In o ma ion
Technology In e na ional Semina (ITIS), 1–7.
h ps://doi.o g/10.1109/ITIS59651.2023.10420330
Ashok, A., Ma co e, É., Zan edeschi, V., Chapados, N., & D ouin, A. (2024). TACTiS-2: Be e ,
Fas e , Simple A en ional Copulas o Mul i a ia e Time Se ies (a Xi :2310.01327). a Xi .
h ps://doi.o g/10.48550/a Xi .2310.01327
Ban, G., Chen, Y., Xiong, Z., Zhuo, Y., & Huang, K. (2024). The uni a ia e model o long- e m
wind speed o ecas ing based on wa ele so h eshold denoising and imp o ed
Au o o me . Ene gy, 290, 130225. h ps://doi.o g/10.1016/j.ene gy.2023.130225
Bou, S., Amagasa, T., & Ki agawa, H. (2022). InT ans: Fas Inc emen al T ans o me o Time
Se ies Da a P edic ion. In C. S auss, A. Cuzzoc ea, G. Ko sis, A. M. Tjoa, & I. Khalil (Eds.),
Da abase and Expe Sys ems Applica ions (pp. 47–61). Sp inge In e na ional Publishing.
h ps://doi.o g/10.1007/978-3-031-12426-6_4
Caldas, F. M., & Soa es, C. (2023). A Tempo al Fusion T ans o me o Long-Te m Explainable
P edic ion o Eme gency Depa men O e c owding. In I. Kop inska, P. Mignone, R. Guido i,
S. Ja oszewicz, H. F öning, F. Gullo, P. M. Fe ei a, D. Roquei o, G. Ceddia, S. Nowaczyk, J.
Gama, R. Ribei o, R. Ga aldà, E. Mascia i, Z. Ras, E. Ri acco, F. Na e o, A. Theissle , P. Biecek,
… S. Pashami (Eds.), Machine Lea ning and P inciples and P ac ice o Knowledge Disco e y in
Da abases (pp. 71–88). Sp inge Na u e Swi ze land. h ps://doi.o g/10.1007/978-3-031-
23618-1_5
Cao, D., Jia, F., A ik, S. O., P is e , T., Zheng, Y., Ye, W., & Liu, Y. (2024). TEMPO: P omp -
based Gene a i e P e- ained T ans o me o Time Se ies Fo ecas ing (a Xi :2310.04948).
a Xi . h ps://doi.o g/10.48550/a Xi .2310.04948
Cao, H., Huang, Z., Yao, T., Wang, J., He, H., & Wang, Y. (2023). InPa o me : E olu iona y
Decomposi ion T ans o me s wi h In e ac i e Pa allel A en ion o Long-Te m Time Se ies
73
Fo ecas ing. P oceedings o he AAAI Con e ence on A i icial In elligence, 37(6), A icle 6.
h ps://doi.o g/10.1609/aaai. 37i6.25845
Cao, Y., & Zhao, X. (2023). Dw o me : Wa ele decomposi ion T ans o me wi h 2D
Va ia ion o Long-Te m Se ies Fo ecas ing. 2023 IEEE 6 h In o ma ion
Technology,Ne wo king,Elec onic and Au oma ion Con ol Con e ence (ITNEC), 6, 1548–
1558. h ps://doi.o g/10.1109/ITNEC56291.2023.10082078
Cen, S., & Lim, C. G. (2024). Mul i-Task Lea ning o he Pa chTCN-TST Model o Sho -Te m
Mul i-Load Ene gy Fo ecas ing Conside ing Indoo En i onmen s in a Sma Building. IEEE
Access, 12, 19553–19568. IEEE Access. h ps://doi.o g/10.1109/ACCESS.2024.3355448
Chen, J., Fan, J., Liu, Z., Xiang, J., & Wu, J. (2023). Seg o me : Segmen -Based T ans o me
wi h Decomposi ion o Long-Te m Se ies Fo ecas ing. 2023 In e na ional Join Con e ence
on Neu al Ne wo ks (IJCNN), 1–8. h ps://doi.o g/10.1109/IJCNN54540.2023.10191412
Chen, P., Zhang, Y., Cheng, Y., Shu, Y., Wang, Y., Wen, Q., Yang, B., & Guo, C. (2023, Oc obe
13). Pa h o me : Mul i-scale T ans o me s wi h Adap i e Pa hways o Time Se ies
Fo ecas ing. The Twel h In e na ional Con e ence on Lea ning Rep esen a ions.
h ps://open e iew.ne / o um?id=lJkOCMP2aW
Chen, Y., Liu, S., Yang, J., Jing, H., Zhao, W., & Yang, G. (2023). A Join Time- equency Domain
T ans o me o Mul i a ia e Time Se ies Fo ecas ing (a Xi :2305.14649). a Xi .
h ps://doi.o g/10.48550/a Xi .2305.14649
Chimmula, V. K. R., & Zhang, L. (2020). Time se ies o ecas ing o COVID-19 ansmission in
Canada using LSTM ne wo ks. Chaos, Soli ons & F ac als, 135, 109864.
h ps://doi.o g/10.1016/j.chaos.2020.109864
Ci s ea, R.-G., Guo, C., Yang, B., Kieu, T., Dong, X., & Pan, S. (2022). T i o me : T iangula ,
Va iable-Speci ic A en ions o Long Sequence Mul i a ia e Time Se ies Fo ecas ing--Full
Ve sion (a Xi :2204.13767). a Xi . h ps://doi.o g/10.48550/a Xi .2204.13767
Dai, T., Wu, B., Liu, P., Li, N., Bao, J., Jiang, Y., & Xia, S.-T. (2023, Oc obe 13). Pe iodici y
Decoupling F amewo k o Long- e m Se ies Fo ecas ing. The Twel h In e na ional
Con e ence on Lea ning Rep esen a ions. h ps://open e iew.ne / o um?id=dp27P5HBB
ddz16. (2024). Ddz16/TSFpape [Compu e so wa e]. h ps://gi hub.com/ddz16/TSFpape
(O iginal wo k published 2022)
Deng, B., Wu, Y., Liu, S., & Xu, Z. (2022). Wind Speed Fo ecas ing o Wind Powe P oduc ion
Based on F equency-Enhanced T ans o me . 2022 4 h In e na ional Con e ence on Machine
Lea ning, Big Da a and Business In elligence (MLBDBI), 151–155.
h ps://doi.o g/10.1109/MLBDBI58171.2022.00036
74
Domínguez-Cid, S., La ios, D. F., Ba bancho, J., Sal ado , A. G., Quin ana-O í, E. S., & León, C.
(2023). TEFNEN: T ans o me o Ene gy Fo ecas ing in Na u al En i onmen . 2023 3 d
In e na ional Con e ence on Elec ical, Compu e , Communica ions and Mecha onics
Enginee ing (ICECCME), 1–6. h ps://doi.o g/10.1109/ICECCME57830.2023.10253223
D ouin, A., Ma co e, É., & Chapados, N. (2022). TACTiS: T ans o me -A en ional Copulas o
Time Se ies (a Xi :2202.03528). a Xi . h p://a xi .o g/abs/2202.03528
Du, D., Su, B., & Wei, Z. (2023). P e o me : P edic i e T ans o me wi h Mul i-Scale Segmen -
Wise Co ela ions o Long-Te m Time Se ies Fo ecas ing. ICASSP 2023 - 2023 IEEE
In e na ional Con e ence on Acous ics, Speech and Signal P ocessing (ICASSP), 1–5.
h ps://doi.o g/10.1109/ICASSP49357.2023.10096881
Eisenach, C., Pa el, Y., & Madeka, D. (2022). MQT ans o me : Mul i-Ho izon Fo ecas s wi h
Con ex Dependen and Feedback-Awa e A en ion (a Xi :2009.14799). a Xi .
h ps://doi.o g/10.48550/a Xi .2009.14799
Fan, J., Wang, B., & Bian, D. (2023). TED o me : Tempo al Fea u e Enhanced Decomposed
T ans o me o Long- e m Se ies Fo ecas ing. IEEE Access, 1–1. IEEE Access.
h ps://doi.o g/10.1109/ACCESS.2023.3287893
Fe nandes, K., Muñoz, A. G., Rami ez-Villegas, J., Agudelo, D., Llanos-He e a, L., Esqui el, A.,
Rod iguez-Espinoza, J., & P age , S. D. (2020). Imp o ing Seasonal P ecipi a ion Fo ecas s o
Ag icul u e in he O inoquía Region o Colombia. h ps://doi.o g/10.1175/WAF-D-19-0122.1
Ga g, L., McClean, S. I., Ba on, M., Meenan, B. J., & Fulle on, K. (2012). In elligen Pa ien
Managemen and Resou ce Planning o Complex, He e ogeneous, and S ochas ic
Heal hca e Sys ems. IEEE T ansac ions on Sys ems, Man, and Cybe ne ics - Pa A: Sys ems
and Humans, 42(6), 1332–1345. IEEE T ansac ions on Sys ems, Man, and Cybe ne ics - Pa
A: Sys ems and Humans. h ps://doi.o g/10.1109/TSMCA.2012.2210211
Ghimi e, S., Nguyen-Huy, T., AL-Musaylh, M. S., Deo, R. C., Casillas-Pé ez, D., & Salcedo-Sanz,
S. (2023). A no el app oach based on in eg a ion o con olu ional neu al ne wo ks and echo
s a e ne wo k o daily elec ici y demand p edic ion. Ene gy, 275, 127430.
h ps://doi.o g/10.1016/j.ene gy.2023.127430
Gong, M., Zhao, Y., Sun, J., Han, C., Sun, G., & Yan, B. (2022). Load o ecas ing o dis ic
hea ing sys em based on In o me . Ene gy, 253, 124179.
h ps://doi.o g/10.1016/j.ene gy.2022.124179
Gong, Z., Tang, Y., & Liang, J. (2023). Pa chMixe : A Pa ch-Mixing A chi ec u e o Long-Te m
Time Se ies Fo ecas ing (a Xi :2310.00655). a Xi .
h ps://doi.o g/10.48550/a Xi .2310.00655
75
Guo, L., Li, R., & Jiang, B. (2021). A Da a-D i en Long Time-Se ies Elec ical Line T ip Faul
P edic ion Me hod Using an Imp o ed S acked-In o me Ne wo k. Senso s, 21(13), A icle
13. h ps://doi.o g/10.3390/s21134466
Hachimi, C. E., Belaqziz, S., Khabba, S., Sebba , B., Dhiba, D., & Chehbouni, A. (2023). Sma
Wea he Da a Managemen Based on A i icial In elligence and Big Da a Analy ics o
P ecision Ag icul u e. Ag icul u e, 13(1), A icle 1.
h ps://doi.o g/10.3390/ag icul u e13010095
Ham, S., Kim, S., Lee, N., Kim, P., Eom, I., Lee, B., Tsai, P.-J., Lee, K., & Yoon, C. (2017).
Compa ison o da a analysis p ocedu es o eal- ime nanopa icle sampling da a using
classical eg ession and ARIMA models. Jou nal o Applied S a is ics, 44(4), 685–699.
h ps://doi.o g/10.1080/02664763.2016.1182132
He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep Residual Lea ning o Image Recogni ion
(a Xi :1512.03385). a Xi . h ps://doi.o g/10.48550/a Xi .1512.03385
Honjo, K., Zhou, X., & Shimizu, S. (2022). CNN-GRU Based Deep Lea ning Model o Demand
Fo ecas in Re ail Indus y. 2022 In e na ional Join Con e ence on Neu al Ne wo ks (IJCNN),
1–8. h ps://doi.o g/10.1109/IJCNN55064.2022.9892599
Huang, J., Ma, M., Dai, Y., Hu, J., & Du, S. (2023). DBAFo me : A Double-B anch A en ion
T ans o me o Long-Te m Time Se ies Fo ecas ing. Human-Cen ic In elligen Sys ems, 3(3),
263–274. h ps://doi.o g/10.1007/s44230-023-00037-z
Huang, S., Zhang, J., He, Y., Fu, X., Fan, L., Yao, G., & Wen, Y. (2022). Sho -Te m Load
Fo ecas ing Based on he CEEMDAN-Sample En opy-BPNN-T ans o me . Ene gies, 15(10),
A icle 10. h ps://doi.o g/10.3390/en15103659
Huang, Y., & Wu, Y. (2023). Sho -Te m Pho o ol aic Powe Fo ecas ing Based on a No el
Au o o me Model. Symme y, 15(1), A icle 1. h ps://doi.o g/10.3390/sym15010238
Jawed, S., & Schmid -Thieme, L. (2022). GQFo me : A Mul i-Quan ile Gene a i e
T ans o me o Time Se ies Fo ecas ing. 2022 IEEE In e na ional Con e ence on Big Da a
(Big Da a), 992–1001. h ps://doi.o g/10.1109/BigDa a55660.2022.10020927
Jiang, Y., Gao, T., Dai, Y., Si, R., Hao, J., Zhang, J., & Gao, D. W. (2022). Ve y sho - e m
esiden ial load o ecas ing based on deep-au o o me . Applied Ene gy, 328, 120120.
h ps://doi.o g/10.1016/j.apene gy.2022.120120
Ji anon, S., Mensin, Y., & Te m i hikun, C. (2023). In elligen Fo ecas ing o Ene gy
Consump ion using Tempo al Fusion T ans o me model. 2023 IEEE In e na ional Con e ence
on Cybe ne ics and Inno a ions (ICCI), 1–5.
h ps://doi.o g/10.1109/ICCI57424.2023.10112297
82
Xie, H., & He, J. (2022). T ans o me Based on Decons uc ion and Do P oduc o Time
Se ies Fo ecas ing o Ai Quali y. 2022 In e na ional Con e ence on Compu e s and A i icial
In elligence Technologies (CAIT), 31–37. h ps://doi.o g/10.1109/CAIT56099.2022.10072078
xiyuanzh. (2024). Xiyuanzh/ ime-se ies-pape s [Compu e so wa e].
h ps://gi hub.com/xiyuanzh/ ime-se ies-pape s (O iginal wo k published 2021)
Yang, Y., & Lu, J. (2023). Fo e o me : An enhanced ans o me -based amewo k o
mul i a ia e ime se ies o ecas ing. Applied In elligence, 53(10), 12521–12540.
h ps://doi.o g/10.1007/s10489-022-04100-3
Yang, Z., Liu, L., Li, N., & Tian, J. (2022). Time Se ies Fo ecas ing o Mo o Bea ing Vib a ion
Based on In o me . Senso s, 22(15), A icle 15. h ps://doi.o g/10.3390/s22155858
Ye, J., Zhao, B., & Liu, D. (2023). Tempo al Decomposi ion T ans o me o P obabilis ic
Ene gy Fo ecas ing. 2022 Fi s In e na ional Con e ence on Cybe -Ene gy Sys ems and
In elligen Ene gy (ICCSIE), 1–5. h ps://doi.o g/10.1109/ICCSIE55183.2023.10175223
Yu, C., Wang, F., Shao, Z., Sun, T., Wu, L., & Xu, Y. (2023). DS o me : A Double Sampling
T ans o me o Mul i a ia e Time Se ies Long- e m P edic ion (a Xi :2308.03274). a Xi .
h ps://doi.o g/10.48550/a Xi .2308.03274
Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2022). A e T ans o me s E ec i e o Time Se ies
Fo ecas ing? (a Xi :2205.13504). a Xi . h ps://doi.o g/10.48550/a Xi .2205.13504
Zeng, P., Hu, G., Zhou, X., Li, S., & Liu, P. (2023). Se o me : A long sequence ime-se ies
o ecas ing model based on bina y posi ion encoding and in o ma ion ans e
egula iza ion. Applied In elligence, 53(12), 15747–15771. h ps://doi.o g/10.1007/s10489-
022-04263-z
Zeng, P., Hu, G., Zhou, X., Li, S., Liu, P., & Liu, S. (2022). Mu o me : A long sequence ime-
se ies o ecas ing model based on modi ied mul i-head a en ion. Knowledge-Based
Sys ems, 254, 109584. h ps://doi.o g/10.1016/j.knosys.2022.109584
Zhang, C., Zhao, L., Yin, Z., & Zhang, Z. (2023). Causal o me : Causal Disco e y-based
T ans o me o Mul i a ia e Time Se ies Fo ecas ing. 2023 16 h In e na ional Cong ess on
Image and Signal P ocessing, BioMedical Enginee ing and In o ma ics (CISP-BMEI), 1–6.
h ps://doi.o g/10.1109/CISP-BMEI60920.2023.10373365
Zhang, X., Jin, X., Gopalswamy, K., Gup a, G., Pa k, Y., Shi, X., Wang, H., Maddix, D. C., &
Wang, Y. (2022). Fi s De-T end hen A end: Re hinking A en ion o Time-Se ies Fo ecas ing
(a Xi :2212.08151). a Xi . h ps://doi.o g/10.48550/a Xi .2212.08151
Zhang, Y., Ma, L., Pal, S., Zhang, Y., & Coa es, M. (2024). Mul i- esolu ion Time-Se ies
T ans o me o Long- e m Fo ecas ing (a Xi :2311.04147). a Xi .
h ps://doi.o g/10.48550/a Xi .2311.04147
83
Zhang, Y., Wu, R., Dascalu, S. M., & Ha is, F. C. (2024). Mul i-Scale T ans o me Py amid
Ne wo ks o Mul i a ia e Time Se ies Fo ecas ing. IEEE Access, 12, 14731–14741. IEEE
Access. h ps://doi.o g/10.1109/ACCESS.2024.3357693
Zhang, Z., Meng, L., & Gu, Y. (2024). SageFo me : Se ies-Awa e F amewo k o Long-Te m
Mul i a ia e Time Se ies Fo ecas ing. IEEE In e ne o Things Jou nal, 1–1. IEEE In e ne o
Things Jou nal. h ps://doi.o g/10.1109/JIOT.2024.3363451
Zhong, Y., Zhou, J., Zhang, T., Yang, J., Li, P., & Deng, D. (2023). NT o me : Nea Time
T ans o me o mul i-s ep p edic ion o wellhead p essu e in ac u ing ope a ions.
Geoene gy Science and Enginee ing, 231, 212407.
h ps://doi.o g/10.1016/j.geoen.2023.212407
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2020). In o me : Beyond
E icien T ans o me o Long Sequence Time-Se ies Fo ecas ing (a Xi :2012.07436). a Xi .
h ps://doi.o g/10.48550/a Xi .2012.07436
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2021). In o me : Beyond
E icien T ans o me o Long Sequence Time-Se ies Fo ecas ing. P oceedings o he AAAI
Con e ence on A i icial In elligence, 35(12), A icle 12.
h ps://doi.o g/10.1609/aaai. 35i12.17325
Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., & Jin, R. (2022). FED o me : F equency Enhanced
Decomposed T ans o me o Long- e m Se ies Fo ecas ing (a Xi :2201.12740). a Xi .
h p://a xi .o g/abs/2201.12740
Zhu, J., Zhao, Z., Zheng, X., An, Z., Guo, Q., Li, Z., Sun, J., & Guo, Y. (2023). Time-Se ies Powe
Fo ecas ing o Wind and Sola Ene gy Based on he SL-T ans o me . Ene gies, 16(22), A icle
22. h ps://doi.o g/10.3390/en16227610
Zhu, S., Zheng, J., & Ma, Q. (2023). MR-T ans o me : Mul i esolu ion T ans o me o
Mul i a ia e Time Se ies P edic ion. IEEE T ansac ions on Neu al Ne wo ks and Lea ning
Sys ems, 1–13. IEEE T ansac ions on Neu al Ne wo ks and Lea ning Sys ems.
h ps://doi.o g/10.1109/TNNLS.2023.3327416
84
APPENDIX A (COMPLEXITY EQUATIONS)
Lowe ing he compu a ional complexi y wi hou sac i icing in o ma ion u iliza ion is a big
challenge, and di e en ans o me s app oach his di e en ly, a i ing a di e en
equa ions.
The anilla ans o me ’s complexi y is 𝒪(𝐿)2 - whe e 𝐿 is he sequence leng h - because
each que y a ends o all keys, equi ing 𝐿×𝐿 compu a ions.
Spa se A en ion Mechanism
(S. Li e al., 2019), in LogSpa se, achie e a complexi y o 𝒪(𝐿(log𝐿)2) h ough a spa se
a en ion mechanism. This mechanism educes he compu a ional load by compu ing
a en ion sco es only o a se o posi ions ha a e loga i hmically spaced.
(H. Zhou e al., 2020), in In o me , each a complexi y o 𝒪(𝐿 log𝐿). I ocuses on on he
mos in o ma i e que ies by using he P obSpa se mechanism which iden i ies a spa se
subse o key-que y pai s wi h he highes a en ion sco es.
(G. Tong e al., 2024), in RSM o me , achie e 𝒪(𝐿 log𝐿), h ough he in oduc ion o he
Residual Spa se A en ion (RSA) mechanism, ha ocuses on he mos dominan que ies, by
le e aging spa se ma ix ope a ions and esidual connec ions.
(W. Li e al., 2023), in Bid o me , each 𝒪((log𝐿)2) h ough he in oduc ion o he
bidi ec ional spa se sel -a en ion mechanism (Bis-A en ion). The Bis-A en ion mechanism
wo ks by spa si ying he sel -a en ion ma ix, ocusing only on he mos ele an pa s o
he inpu sequence.
F equency Domain and Fou ie -based Me hods
(H. Wu e al., 2021), in Au o o me , achie e a complexi y o 𝒪(𝐿 log𝐿) by le e aging he
Fas Fou ie T ans o m (FFT) o compu e he au oco ela ion e icien ly. The p ocess in ol es
calcula ing he au oco ela ion o he ime se ies o iden i y he mos signi ican pe iodic
lags, agg ega ing simila sub-se ies e icien ly.
(T. Zhou e al., 2022), in FED o me , each a complexi y o 𝒪(𝐿) by ans o ming he inpu
da a in o he equency domain using Fou ie and wa ele ans o ms and hen p ocessing
only a ixed numbe o selec ed equency componen s.
(Ouyang e al., 2023a), in Rank o me , a i e a 𝒪(𝐿 log𝐿), by in eg a ing he Fas Fou ie
T ans o m (FFT) and he Wiene -Khinchin heo em o calcula ing he Au oco ela ion
Func ion (ACF) o anked ime se ies da a.
(Y. Wang e al., 2023), in DEST o me , achie e 𝒪(𝐿) by employing he Fas Fou ie T ans o m
(FFT) is used o decompose he ime se ies da a in o seasonal and end componen s, and by
85
u he sampling he equency componen s and applying he specialized Mul i-View and
Mul i-Scale A en ion mechanisms.
Pa ch-based and Segmen a ion App oaches
(Ci s ea e al., 2022), in T i o me , achie e linea complexi y, 𝒪(𝐿). This is achie ed h ough
he in oduc ion o he "Pa ch A en ion" mechanism, di iding he ime se ies in o smalle
pa ches, and wi hin each pa ch, a pseudo imes amp is used o compu e a en ion sco es.
This educes he numbe o a en ion compu a ions needed wi hin each pa ch o linea
complexi y. Addi ionally, he T i o me employs a iangula s uc u e o s acking mul iple
laye s o Pa ch A en ion, whe e he size o each subsequen laye dec eases exponen ially,
hus main aining he o e all linea complexi y o he en i e model.
(Nie e al., 2023), in Pa chTST, achie e 𝒪(𝐿/𝑆)2), by segmen ing he inpu ime se ies in o
pa ches, e ec i ely educing he numbe o inpu okens 𝐿 by a ac o ela ed o he pa ch
leng h 𝑃 and s ide 𝑆, signi ican ly educing he compu a ional load and memo y
equi emen s when 𝑆>1.
(J. Chen e al., 2023), in Seg o me , each 𝒪( (𝐿𝑙)2), whe e 𝑙 is he segmen leng h, by
di iding he en i e sequence in o smalle segmen s o leng h 𝑙 and ocusing he a en ion
mechanism on hese segmen s a he han he en i e sequence.
(Lee e al., 2024), in TS-Fas o me , educe he complexi y o 𝒪( (𝐿𝑙)2), by segmen ing he
inpu sequence in o smalle sub-windows o leng h 𝑙.
(Shen e al., 2023), in FPP o me , each 𝒪(𝐿×𝑃) de e mined by i s use o pa ch-wise and
elemen -wise a en ion mechanisms, whe e 𝑃 is he size o each pa ch.
Hie a chical and Mul i-scale A en ion Mechanisms
(Tang & Ma eson, 2021), in P oT an, achie e a complexi y o 𝒪(𝐿2 𝑑), whe e 𝑇 is he leng h
o he ime se ies and 𝑑 is he dimensionali y o he la en space. This complexi y a ises om
he use o he a en ion mechanisms ha in ol es ope a ions on ma ices o size 𝑇×𝑑.
Each a en ion ope a ion equi es calcula ing pai wise in e ac ions be ween all ime s eps,
leading o a quad a ic dependence on he sequence leng h 𝑇. Addi ionally, he hie a chical
s uc u e wi h mul iple laye s o la en a iables adds a linea ac o o he complexi y.
(S. Liu e al., 2022), in Py a o me , achie e linea complexi y, 𝒪(𝐿), by limi ing he scope o
a en ion o local neighbo hoods and summa izing in o ma ion a p og essi ely coa se
le els. This is done h ough i s hie a chical and py amidal s uc u e, which employing a
spa se a en ion mechanism ac oss mul iple scales. Each node in his s uc u e a ends only
o a selec ed subse o o he nodes, bo h wi hin i s own scale and ac oss scales.
86
Simpli ied and Local A en ion Mechanisms
(D ouin e al., 2022), in TACTiS, each a complexi y o 𝒪(𝑛2∙ 𝑙𝑚𝑎𝑥 + 𝑛 ∙ 𝑙𝑚𝑎𝑥2), whe e n
ep esen s he numbe o di e en ime se ies and 𝑙𝑚𝑎𝑥 is he leng h o he longes ime
se ies. This complexi y is achie ed h ough he model’s s a egic use o a en ion
mechanisms ac oss di e en laye s; a en ion is applied in wo dimensions - ac oss di e en
a iables and ac oss di e en ime s eps wi hin each a iable.
(Bou e al., 2022), in InT ans, each 𝒪(𝑆), by imp o ing compu a ional e iciency wi h
inc emen al compu a ions, whe e S is he leng h o he non-o e lapping pa o he inpu ,
educing edundan ope a ions.
(Y. Li, Lu, e al., 2023), in Con o me , each 𝒪(𝐿) by in oducing he sliding-window a en ion
mechanism, whe e each elemen only a ends o a ixed numbe o nea by elemen s wi hin
a p ede ined window, a he han he en i e sequence.
(Y. Yang & Lu, 2023), in Fo e o me , each 𝒪(𝐿2∙ 𝑑), whe e 𝑑 is he dimensionali y o each
inpu ec o . This is due o he sel -a en ion mechanism, whe e each elemen in he
sequence a ends o e e y o he elemen , esul ing in an 𝐿×𝐿 ma ix o a en ion sco es.
The e o e, he o al compu a ional complexi y o he a en ion mechanism becomes 𝒪(𝐿2∙
𝑑), e lec ing he quad a ic ela ionship wi h he sequence leng h and he linea
ela ionship wi h he dimensionali y o he da a.
Discussion
Spa se a en ion mechanisms signi ican ly cu down compu a ional complexi y by ocusing
a en ion only on he mos ele an pa s o he inpu sequence. F equency-domain
me hods u ilize Fas Fou ie T ans o m (FFT) and ela ed echniques o ans o m and
p ocess da a e icien ly. Pa ch-based and segmen a ion app oaches b eak down sequences
in o smalle segmen s o pa ches, educing he numbe o compu a ions equi ed.
Hie a chical and mul i-scale a en ion mechanisms achie e e iciency by o ganizing a en ion
hie a chically ac oss di e en scales o laye s. Las ly, simpli ied and local a en ion
mechanisms ocus on local neighbou hoods o inc emen al compu a ions o s eamline he
a en ion p ocess. This end owa ds di e se, specialized me hods indica es a obus e o
o balance compu a ional e iciency wi h e ec i e in o ma ion u iliza ion in ans o me
a chi ec u es.
87
APPENDIX B (PAPERS’ STUDY EXTRA)
Figu e 0.1 – Ba plo o he numbe o pages o he pape s, his og am o he numbe o
pages in he annexes, his og am o he numbe o e e ences
Mos pape s ha e a ound 10 pages, 79 o he pape s ha e no annexes and he ones ha do
mos ly ha e om 4 o 10 pages – he annexes mos ly ocus on p oo s, sensi i i y s udies,
abla ion s udies and de ails abou aining, uning, and he da ase s. The Dis ibu ion o he
numbe o e e ences is cen e ed a ound 40.
56 o he pape s we e published in con e ence p oceedings and he 46 emaining ones we e
published in books. The mos common con e ences we e ICLR, Neu IPS, AAAI, CIKM, ACML,
IJCAI, IJCNN, and 27 unique con e ences. The mos common books we e Applied
In elligence, IEEE Access and o he IEEE books, Ene gy, Senso s, and 13 unique books.
The au ho s o each pape we e e ie ed and hen analysed. 22 o he au ho s ha e
con ibu ed o 2 o he pape s, and he emaining 392 only con ibu ed o one.
Figu e 0.2 - Ba Plo o he Au ho s coun pe Pape ; Dis ibu ion o he Au ho s Based on he
Coun y
88
The e a e only 3 pape s wi h a single au ho , mos a e published h ough a collabo a ion,
bu eams o mo e hen se en a e a e. This migh sugges ha he esea ch is oo complex
o be done alone and ha coo dina ing wi h mo e han 6 people becomes inc easingly
complex.
Mo e han hal o he Au ho s a e in China, wi h a signi ican amoun in he USA, and he
es dis ibu ed among o he 21 coun ies.
89
APPENDIX C (TABLE OF SUMMARY OF CHANGES)
Table 0.1 has he lis o he analysed ans o me s, he e e ence o he espec i e pape , i
he code is publicly a ailable, he summa y o he changes in key-wo ds and he
mo i a ions/ issues add essed by he p oposed ans o me .
Table 0.1 – Summa y o he Changes in he Analysed T ans o me s, in ch onological o de o
publica ion
T ans o me
Re e ence
Code
Summed up changes
Issues
Add essed and
Mo i a ions
Log ans/Logspa se
(S. Li e al., 2019)
Yes
Spa se Con olu ional Sel
A en ion + Rolling Window
Locali y-Agnos ic
+ Memo y
Bo leneck
TFT
(Lim e al., 2020)
Yes
S a ic Co a ia e Encode s
(GRN) + Ga ing Mechanisms
+ Sequence- o-Sequence
Laye + In e p e able Mul i-
Head A en ion + Tempo al
Fusion Decode + Va iable
Selec ion Ne wo ks +
Quan ile P edic ions
In e p e abili y +
Co a ia es
AST
(S. Wu e al.,
2020)
Yes
Co a ia es + α-en max
Spa se A en ion +
Ad e sa ial aining + GAN
amewo k
E o
Acumula ion due
o Au o-
Reg ession +
P obabilis ic
Fo ecas ing
In o me
(H. Zhou e al.,
2020)
Yes
P obSpa se sel -a en ion (KL
Di e gence)+ Gene a i e
Decode + Sel -a en ion
dis illing (Max Pooling)
Memo y
Bo leneck +
Speed Plunge
S acked-In o me
Ne wo k
(Guo e al., 2021)
No
S acked S uc u e +
P obabilis ic Spa se A en ion
+ Sel -A en ion Dis illing +
G adien Cen aliza ion and
Adam Op imize + Gene a i e
S yle Decode
Accu acy and
Speed on PV
Fo ecas ing
TCCT
(Shen & Wang,
2022)
Yes
CPSA en ion + Sel -a en ion
Dis illing (Dila ed Casual
Con olu ion) + Pass h ough
Mechanism
Memo y
Bo leneck +
Insensi i i y o
Local Con ex +
Exis ing CNN
hyb ids a e
Loosely-Coupled
90
AGCNT
(H. Su e al.,
2021)
No
P obSpa se Adap i e G aph
Sel -A en ion + Dis illing +
Gene a i e In e ence +
Con olu ion
P obspa se
Des oys
Rela ionships
Be ween
Sequences –
Doesn’ Cap u e
Co ela ion
Be ween
Sequences
SSDNe
(Y. Lin e al.,
2021)
No
Co a ia es + SSM + NLL and
MAE Loss Func ion +
P obabilis ic Fo ecas ing
In e p e abili y +
In e es in
Kalman Fil e
wi h he
P oblem o Being
Compu a ionally
Expensi e
Au o o me
(H. Wu e al.,
2021)
Yes
Decomposi ion Block + Au o-
Co ela ion Inne Block
Memo y
Bo leneck +
Cap u ing
In ica e
Tempo al
Pa e ns
P oT an
(Tang &
Ma eson, 2021)
No
SSM + P obabilis ic Modeling
+ S ochas ic La en Va iables
+ Va ia ional In e ence + KL
Di e gence and Va ia ional
In e ence Loss Func ion
Cap u ing Non-
Ma ko ian
Dynamics and
Non-Linea
In e ac ions
Py a o me
(S. Liu e al.,
2022)
Yes
Py amidal A en ion Model
(Spa se) + Coa se -Scale
Cons uc ion Module
To Build a
Flexible bu
Pa simonious
Model ha Can
Cap u e a Wide
Range o
Tempo al
Dependencies
CEEMDAN-Sample
En opy-BPNN-
T ans o me
(S. Huang e al.,
2022)
No
Comple e Ensemble
Empi ical Mode
Decomposi ion wi h Adap i e
Noise + Sample En opy +
Back P opaga ion Neu al
Ne wo k
Powe Load
Being S ochas ic
FED o me
(T. Zhou e al.,
2022)
Yes
F equency Enhanced Block +
Disc e e Wa ele T ans o m
+ F equency Enhanced
A en ion + Mix u e O
Expe Decomposi ion
E iciency +
Unable o
Cap u e T end
TACTIS
(D ouin e al.,
Yes
Co a ia es + imes amps +
Es ima ing he
91
2022)
Time Se ies Tokens + Non-
Pa ame ic Copula +
A en ional Copula
Mechanism
Join P edic i e
Dis ibu ion o
High-
dimensional
Mul i a ia e
Time Se ies +
Missing Values
ACT
(S. Wu e al.,
2020)
No
Con olu ional A en ion
Block + Decoding Mode +
Ad e sa ial aining
E o
Accumula ion +
Insensi i e o
Local Con ex +
No Modelling
Dis ibu ion o
Da a
T i o me
(Ci s ea e al.,
2022)
Yes
Pa ch A en ion (Pa ches and
Pseudo Times amps) +
Recu ence + T iangula
S acking + Va iable Speci ic
model
Memo y
Bo leneck +
Di icul y
Cap u ing
Di e en
Va iables’
Tempo al
Dynamics
SWLHT
(Y. Liu e al.,
2022)
Yes
O e lapping Segmen s +
Memo y-E icien Module +
Enhanced A en ion
Mechanism + Linea
Au o eg essi e Module +
I e a i e Conca ena ion o
Single-S ep Fo ecas s
Complex and
Non-linea
In e dependence
Be ween Time
S eps and
Di e en
Va iables +
P edic ion
F agmen a ion +
Insensi i i y o
Da a Scale + Lack
o Da a T aining
InT ans
(Bou e al., 2022)
No
Inc emen al Posi ional
Embedding + Inc emen al
Tempo al Embedding +
Inc emen al Value
Embedding (Con olu ion) +
Inc emen al Sel -a en ion
In o me Being
Unable o
Inc emen ally
Lea n + Long
Time Se ies
Fo ecas ing
Y o me
(Madhusudhanan
e al., 2022)
Yes
Downscaling + Upscaling +
Con ac ing P obSpa se Sel -
A en ion Blocks + Expanding
P obSpa se C oss-A en ion
Block + Auxilia y
Recons uc ion Loss
Di icul ies in
Cap u ing Long-
Range
Dependencies
E ec i ely +
Spa se
A chi ec u e
98
Fo ecas ing
Ra he Than
Indi idual Load
Fo ecas ing,
Which Canno
Suppo
Con olled
Demand
Response
P og ams
SL-T ans o me
(J. Zhu e al.,
2023)
No
Sa i zky–Golay (SG) Fil e +
Local Ou lie Fac o (LOF)
il e + LSTMs
Noise and
Tempo al
In icacies
FPP o me
(Shen e al.,
2023)
Yes
Diagonal-Masked Sel -
A en ion + Combined Pa ch-
wise and Elemen -wise
A en ion + Hie a chical
Fea u e Mapping +
Composi e Loss Func ion
Ex ac ing he
Fea u es o Inpu
Sequence and
Seeking he
Rela ions o
Inpu and
P edic ion
Sequence
BasisFo me
(Ni e al., 2024)
Yes
Basis Vec o s + Coe Module
+ Bidi ec ional C oss-
A en ion + Regula iza ion
Te m + Dual Loss Func ion +
End- o-End T aining
Using Bases
TCLN
(S. Ma e al.,
2023)
No
Mul i-ke nel CNN Module +
LSTM Encode +
Au o eg essi e Componen
Lack o Dep h in
Ex ac ing Spa ial
and
Spa io empo al
Fea u es
Be ween
Va iables
NT o me
(Zhong e al.,
2023)
No
Lea nable Times amp
Encoding + Tempo al and
Non- empo al P ocessing +
Tempo al A en ion +
Tempo al Dimension
Co ela ion + Fea u e
Con e sion Mechanism +
Loss Func ion wi h L2
Regula iza ion
Da ase
HAFF
(Z. Wang e al.,
2023)
No
Re ine Au o-Co ela ion
Mechanism o Au o o me
Di icul o
Fo ecas he
Elec ici y
Demand Se ies
Because o he
99
In luence o
Cyclical Fac o s
and Small Da a
Amoun s.
MR-T ans o me
(S. Zhu e al.,
2023)
No
Co a ia es + Adap i e
Segmen a ion (DWT) +
Va iable-Speci ic Tempo al
Con olu ion + Va iable-
Speci ic A en ion + Long
Sho -Te m A en ion
Con ex -agnos ic
Fo wa d o me
(Qu e al., 2024)
No
Inpu Rep esen a ion (Value,
Times amp and In o ma ion
Embeddings) and + Mul i-
Scale Fo wa d Sel -A en ion
+ No el Loss Func ion
Igno e Special
E en s
RSM o me
(G. Tong e al.,
2024)
No
Up-and-down Sampling +
Residual Spa se A en ion
Mechanism + C oss-Scale
Fea u e Rela ionships+
Adap i e Hube Loss
Func ion
Cons ained by
Single Time
Scale, he
Quad a ic
Calcula ion
Complexi y o
he Sel -
A en ion
Mechanism, and
he High
Memo y
Occupa ion.
Mul i a e-Fo me
(D. Liu e al.,
2024)
No
Da a Chunking + Coa se-
G ained Da a
Complemen a ion (CNN) +
Sel -Recons uc ion E o +
Sampling-Type Coding + Fine-
G ained Complemen a ion +
Unsupe ised P e aining
Ubiqui ous
Mul i a e
Sampling
Cha ac e is ics o
he Da a
SageFo me
(Z. Zhang e al.,
2024)
Yes
Global Tokens + Se ies-Awa e
F amewo k + G aph-
enhanced T ans o me +
Spa si y + Fo ecas ingHead
P e ailing
Me hods Ei he
Ma ginalize
In e -se ies
Dependencies o
O e look Them
En i ely
MTPNe
(Y. Zhang, Wu, e
al., 2024)
Yes
Inpu Rep esen a ion
(Dimension-In a ian
Embedding + CNN + Pa ches)
+ Mul i-scale T ans o me
Py amid Ne wo k +
Hie a chical In e -Scale
P io Wo k has
Been Con ined o
Modeling
Tempo al
Dependencies a
Ei he a Fixed
100
Connec ions
Scale o Mul iple
Scales ha
Exponen ially
Inc ease (Mos
wi h Base 2).
This Limi a ion
Impedes Thei
Capaci y o
E ec i ely
Cap u e Di e se
Seasonali ies.
Pa chTCN-TST
(Cen & Lim,
2024)
No
Segmen s (TCN) + Channel-
Independen + TCN Residual
Blocks + Ins ance
No maliza ion and De-
No maliza ion + Mul i-Task
Lea ning
E iciency
G aph o me
(Y. Wang e al.,
2024)
No
Lea nable Tokens (GNsNs) +
G aph Sel -A en ion
Mechanism (Spa se) +
Dila ed Causal Con olu ion +
Tempo al Ine ia Module +
Mul i-scale Fea u e Fusion
Canno
E ec i ely
Exploi he
Po en ial Spa ial
Co ela ion
Be ween
Va iables
TS-Fas o me
(Lee e al., 2024)
Yes
Sub Window Tokenize + P e-
ained Encode + Pas
A en ion Decode
Sequen ial
Rela ionships +
Slow Speed
MASTER
(T. Li, Liu, e al.,
2023)
Yes
Ma ke -Guided Ga ing
Mechanism + Momen a y
and C oss-Time S ock
Co ela ion + Tempo al
Agg ega ion
Fi s , S ock
Co ela ions
O en Occu
Momen a ily and
in a C oss- ime
Manne . Second,
he Fea u e
E ec i eness is
Dynamic wi h
Ma ke
Va ia ion, Which
A ec s Bo h he
S ock Sequen ial
Pa e ns and
Thei
Co ela ions.
MOEA
(Ban e al., 2024)
No
In eg a e Mix u e O Expe
Decomposi ion and Wa ele
So Th eshold Denoising
Da ase
PWD o me
(Z. Wang e al.,
No
Posi ion Weigh s + F equency
T end o Reduce
101
2024)
Selec ion Module +
De o mable-Local
Agg ega ion Mechanism
(Adap i e Windows)
Compu a ional
Complexi y a
he Expense o
Time
In o ma ion
Agg ega ion
Capabili y +
O de
in o ma ion loss
Hid o me
(Z. Liu e al.,
2024)
No
Segmen -and-Me ge
A chi ec u e + Replacemen
o Mul i-Head A en ion wi h
Recu ence and Linea
A en ion
Pe mu a ion-
in a ian +
Missing Mul i-
scale Local
Fea u es and
In o ma ion
om he
F equency
Domain
JTFT
(Y. Chen e al.,
2023)
Yes
Lea nable Posi ional
Encoding (Join Time-
F equency Domain
Rep esen a ion + Cus omized
Disc e e Cosine T ans o m) +
Low-Rank A en ion Laye
E iciency +
Mul i a ia e
Fo ecas ing
MTST
(B. Li, Cui, e al.,
2023)
Yes
Rela i e Posi ional Encoding
+ Pa ch-Le el Tokeniza ion +
Mul i- esolu ion p ocessing +
Mul i Scale Fea u e Lea ning
+ Adap ed A en ion
Mechanism
Pa ch Size
PDF
(Dai e al., 2023)
Yes
Mul i-pe iodic Decoupling
Block (FFT and
Segmen a ion) + Dual
Va ia ions Modeling Block
(CNN) + Va ia ions
Agg ega ion Block
In ica e
Tempo al
Pa e ns
Pa h o me
(P. Chen e al.,
2023)
Yes
Mul i-Scale Rou e (Adap i e
Segmen s) + Adap i e
Pa hways + Dual A en ion
Challenging o
Cap u e
Di e en
Cha ac e is ics
Spanning Va ious
Scales
TACTIS-2
(Ashok e al.,
2024)
Yes
Dual-encode (Ma ginal and
Copula Dis ibu ions o
Co a ia es and Inpu
Rep esen a ion) + Causal
A en ion Mechanism + Two-
Inc ease
Flexibili y
102
S age Op imiza ion P ocess +
Copulas
TEMPO
(D. Cao e al.,
2024)
No
Specialized Embeddings +
Ze o-sho Lea ning (So
P omp ing Mechanism) +
Gene a i e P e- ained
T ans o me (GPT)
Explo e Whe he
GPT- ype
A chi ec u es
Can Be E ec i e
o Time Se ies
103
APPENDIX D (AGGREGATION OF RESULTS)
In his Sec ion, he inal esul s om di e en pape s a e agg ega ed, bu i is no possible o
do his o he main esul s.
Table 0.1 – E olu ion o In o me me ics h oughou di e en pape s
Me ic
In o me
In o me in
Linea
In o me in
Pa chTST
In o me in
Con o me
In o me in
DWT o me
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Elec ici y/ ECL
96
0,274
0,368
0,304
0,393
0,5423
0,5568
2,26
0,368
192
0,296
0,386
0,327
0,417
0,5304
0,5549
0,284
0,384
336
0,3
0,394
0,333
0,422
0,6429
0,5921
0,334
0,423
720
0,373
0,439
0,351
0,427
0,9534
0,7789
0,592
0,589
Exchange
96
0,847
0,752
0,847
0,752
0,231
0,3841
0,395
0,453
192
1,204
0,895
1,204
0,895
0,3079
0,4488
0,692
0,6
336
1,672
1,036
1,672
1,036
0,5902
0,6306
0,809
0,649
720
2,478
1,31
2,478
1,31
0,863
0,7953
1,111
0,769
T a ic
96
0,719
0,391
0,733
0,41
0,739
0,413
192
0,696
0,379
0,777
0,435
0,762
0,424
336
0,777
0,42
0,776
0,434
0,862
0,487
720
0,864
0,472
0,827
0,466
1,056
0,53
Wea he
96
0,3
0,384
0,354
0,405
0,3929
0,3231
192
0,598
0,544
0,419
0,434
0,4396
0,4332
336
0,578
0,523
0,583
0,543
0,5848
0,5197
720
1,059
0,741
0,916
0,705
0,7051
0,5881
ILI
24
5,764
1,677
4,657
1,449
36
4,755
1,467
4,65
1,463
48
4,763
1,469
5,004
1,542
60
5,264
1,564
5,071
1,543
ETTh1
96
0,865
0,713
0,941
0,769
0,8901
0,6498
0,905
0,751
192
1,008
0,792
1,007
0,786
1,0463
0,7082
1,006
0,786
336
1,107
0,809
1,038
0,784
1,1237
0,7383
1,036
0,783
720
1,181
0,865
1,144
0,857
1,1047
0,7292
1,155
0,846
ETTh2
96
3,755
1,525
1,549
0,952
2,833
1,321
192
5,602
1,931
3,792
1,542
2,625
2,091
336
4,721
1,835
4,215
1,642
5,309
1,953
720
3,647
1,625
3,656
1,619
3,989
1,665
ETTm1
96
0,672
0,571
0,626
0,56
1,0921
0,7023
0,623
0,559
192
0,795
0,669
0,725
0,619
1,2657
0,7898
0,854
0,686
336
1,212
0,871
1,005
0,741
1,3849
0,8459
1,074
0,769
720
1,166
0,823
1,133
0,845
1,3537
0,8492
1,236
0,826
ETTm2
96
0,365
0,453
0,355
0,462
0,387
0,478
192
0,533
0,563
0,595
0,586
0,808
0,706
336
1,363
0,887
1,27
0,871
1,464
0,928
720
3,379
1,338
3,001
1,267
3,904
1,466
104
In Table 0.1 – E olu ion o In o me me ics h oughou di e en pape s, he 1s column was
in oduced in he In o me pape (H. Zhou e al., 2021). The 2nd column was in oduced in
he Linea pape (A. Zeng e al., 2022), and using he same hype -pa ame e s ain he model
o he missing da ase s. The 3 d column was in oduced in he Pa chTST pape (Nie e al.,
2023), changing he size o he look-back window. The 4 h column was in oduced in he
Con o me model (Y. Li, Lu, e al., 2023), se ing he sampling ac o o 1. The 5 h column was
in oduced in he DWT o me pape (Y. Cao & Zhao, 2023), using he same pa ame e s as
he i s column. The alues om hese columns we e copied in o he pape s.
This shows he inhe en s ochas ici y associa ed wi h ans o me s by compa ing he i s
and las column, as well as he impac o using di e en pa ame e s in he pe o mance o
he model ha was done in he 3 d and 4 h columns.
Table 0.2 and Table 0.3 agglome a e he esul s o pape s ha use R0.5 and CRPS. As said
be o e, mos o he pape s use he MSE and MAE me ics. Fo mul i a ia e esul s, he e a e
2 main o ecas ing windows {24, 48, 168, 336, 720} and {96, 192, 336, 720}. The i s has 42
columns and 31 ows while he second has 82 columns and 38 ows. The same 2 o ecas ing
windows a e used o uni a ia e o ecas ing ha ing 14 columns and 27 ows, and 36 columns
and 37 ows, espec i ely. This means ha he ables canno be shown he e, bu hey a e in
excel shee “Agglome a ion”.
Table 0.2 – Agglome a ion o esul s ha use R0.5 as he me ic
R0.5
Log ans
ARIMA
ETS
TRMF
DeepAR
DeepS a e
TFT
AST
In o me
ACT
PDT ans
Elec ici y-
c 1d
0.059
0.154
0.101
0.084
0.075
0.083
0.055
0.042
±
0.007
0.062 ±
0.005
0.040
±
0.007
0.058
Elec ici y-
c 7d
0.070
0.283
0.121
0.087
0.082
0.085
0.057
±
0.010
0.072 ±
0.009
0.055
±
0.010
0.068
T a ic-c
1d
0.122
0.223
0.236
0.186
0.161
0.167
0.095
0.093
±
0.010
0.120 ±
0.019
0.091
±
0.010
0.113
T a ic-c
7d
0.139
0.492
0.509
0.202
0.179
0.168
0.125
±
0.019
0.130 ±
0.023
0.119
±
0.019
0.126
105
Table 0.3 - Agglome a ion o esul s ha use CRPS as he me ic
CRPS-Sum
P oT an
Au o-
ARIMA
ETS
TimeG ad
GPVa
TACTiS
TACTiS-2
SOLAR
0.194 ±
0.030
ELECTRICITY
0.016 ±
0.001
0.077 ±
0.016
0.059 ±
0.011
0.067 ±
0.028
0.035 ±
0.011
0.021 ±
0.005
0.020 ±
0.005
TRAFFIC
0.028 ±
0.001
TAXI
0.084 ±
0.003
WIKIPEDIA
0.047 ±
0.004
ed-md
0.043 ±
0.005
0.037 ±
0.010
0.094 ±
0.030
0.067 ±
0.008
0.042 ±
0.009
0.035 ±
0.005
kdd-cup
0.625 ±
0.066
0.408 ±
0.030
0.326 ±
0.024
0.290 ±
0.005
0.237 ±
0.013
0.234 ±
0.011
sola -10min
0.994 ±
0.216
0.678 ±
0.097
0.540 ±
0.044
0.254 ±
0.028
0.311 ±
0.061
0.240 ±
0.027
a ic
0.222 ±
0.005
0.353 ±
0.011
0.126 ±
0.019
0.145 ±
0.010
0.071 ±
0.008
0.078 ±
0.008
106