BEYOND NOTATION: A DIGITAL PLATFORM FOR TRANSCRIBING
AND ANALYZING ORAL MELODIC TRADITIONS
Jona han Mye s Da d Neuman
Depa men o Music, Uni e si y o Cali o nia, San a C uz, San a C uz, CA, USA
{JBMye s, DNeuman}@ucsc.edu
ABSTRACT
This pape desc ibes he In e ac i e Digi al T ansc ip ion
and Analysis Pla o m (IDTAP): The IDTAP is a
web-based applica ion designed o enable use s o
digi ally ansc ibe, a chi e, sha e, and analyze audio
eco dings o o al melodic adi ions. The pla o m’s
unde lying music- heo e ical p emises and co esponding
da a a chi ec u e ha e ini ially been de eloped o align
wi h he idioma ic ea u es o Hindus ani music (i.e.,
No h Indian classical music). These ea u es necessi a e
a lexible a ay o pi ch-con ou cu es; adjus able uning
sys ems ha allow o he ep esen a ion o a ange
o mic o onal con igu a ions ound in bo h his o ical
and con empo a y p ac ice; exp essi e mic o onal pi ch
in lec ions be ween uning sys em pi ches; and a highly
p ecise hy hmic ep esen a ion ha cap u es sub le
mic o- iming nuances, exp essi e empo a ia ions, and
bo h me ic and non-me ic hy hmic s uc u es exac ly as
pe o med. The IDTAP’s a chi e, ansc ip ion edi o , and
analysis sui e join ly a e designed o u u e expansion o
include a ange o musical adi ions, opening mul iple
sound collec ions and a chi es o digi al p ese a ion,
pedagogy, and app ecia ion, as well as s a is ical,
quan i a i e, and in e p e i e analysis. The pla o m and
co esponding da a a chi ec u e equips schola s om a
ange o disciplina y backg ounds o apply he powe
o wen y- i s -cen u y compu a ional me hodologies and
la ge da ase s o humanis ic and c ea i e endea o s.
1. INTRODUCTION
The IDTAP is an open-sou ce and mul i-laye ed in e ac i e
digi al a chi e o o al melodic exp essi e adi ions. 1
The pla o m allows use s o upload audio eco dings,
ansc ibe melodic sound in ui i ely, e icien ly, and
accu a ely om hose eco dings; es he accu acy o
hei ansc ip ion h ough syn hesized audio playback;
1The IDTAP is eleased unde he MIT licence. Sou ce code
is a ailable a h ps://gi hub.com/jon-mye s/id ap. The
pla o m is a ailable a h ps://swa a.s udio
© J. Mye s and D. Neuman. Licensed unde a
C ea i e Commons A ibu ion 4.0 In e na ional License (CC BY 4.0).
A ibu ion: J. Mye s and D. Neuman, “Beyond No a ion: A Digi al
Pla o m o T ansc ibing and Analyzing O al Melodic T adi ions”, in
P oc. o he 26 h In . Socie y o Music In o ma ion Re ie al Con .,
Daejeon, Sou h Ko ea, 2025.
and analyze he ansc ip ions quali a i ely, quan i a i ely,
and compu a ionally. The b oade goal o he IDTAP is
o de-cen e music schola ship and composi ion om a
wes e n heo y and no a ional amewo k while cen e ing
he place o music (o melodic exp essi e adi ions) in
humanis ic and social scien i ic s udies. S a no a ion has
long been he p ima y ool o g aphically ep esen ing
music in academia wo ld-wide, and MIDI emains he
dominan da a o ma used in MIR and compu a ional
music analysis con ex s. Bo h s a no a ion and MIDI
inhe i and ein o ce he heo e ical p emises o Wes e n
music heo y, pa icula ly he disc e iza ion o pi ch and
hy hm in o quan ized and equally empe ed ca ego ies.
This limi a ion makes hem ill-sui ed o accu a ely
ep esen a ious aspec s o musical in o ma ion such as
in ica e glissandi, idioma ic uning sys ems, and lexible
in ona ion ound in many non-wes e n music adi ions.
Since he 1950s, schola s in musicology and
e hnomusicology ha e a gued agains he use o s a
no a ion o he ansc ip ion, composi ion, and analysis
o non-Eu opean a music adi ions [1–4]. Despi e
his, many esea che s and a is s con inue o use i
when engaging wi h non-wes e n epe oi es. This
p esen s p oblems ha a e e hical, empi ical, and
epis emic. E hically, s a no a ion imposes a Eu ocen ic
ep esen a ion o musical o m. Empi ically, s a no a ion
la ens ou impo an aspec s o many o al-melodic
idioms. Epis emically, i discou ages schola s wi hou
knowledge o s a no a ion, le alone o he me hodological
ools o include melody and hy hm in hei esea ch. This
e ec i ely emo es om humanis ic and social scien i ic
inqui y a cen al aspec o e e yday li e— he ela ionship
o o ganized sound o a ange o communi ies, be
hey small scale p oxima e communi ies o la ge-scale
eligious, dynas ic, o na ional imagined communi ies.
These ela ionships a e e en mo e elusi e in o al cul u es
and hei secula and spi i ual exp essi e adi ions.
By con as , he IDTAP p o ides a mul i-laye ed,
in e ac i e pla o m and co esponding s o e o knowledge,
se ing as a ansdisciplina y esea ch a chi e ha b idges
compu a ional media, linguis ics, compa a i e li e a u e,
s a is ics, his o y, olklo e, eligious s udies, cul u al and
compa a i e musicology, e hnomusicology, and music
composi ion. The goal o he IDTAP esea ch eam is
o expand he pla o m so ha eco ded music om a
di e si y o o al cul u es can be bo h p ese ed and made
a ailable o music-making and esea ch in empi ically
407
analyzable and compa able da a o ma s.
The IDTAP so wa e ep esen s a di e si y o melodic
con ou s h ough ou eimagining o a widely held
music- heo e ical ene . Ra he han ea ing a ixed pi ch
no e as he a omic uni o musical s uc u e, he IDTAP is
o ganized a ound a succession o “ ajec o ies”: o mally
speci ied a che ypal pa hs be ween pi ches, ac oss a se ies
o pi ches, o on a ixed pi ch. These ajec o ies can
ep esen a ange o inely calib a ed glissandi ha appea
in many musical adi ions ou side o he keyboa d and
s a -o ien ed Eu ocen ic musical pa adigm.
T ajec o ies p o ide a middle-le el ep esen a ion akin
o an alphabe o melodic mo ions. Fundamen al
equency da a alone, o en used di ec ly in analyses,
ends o be excessi ely g anula and edundan , o e ing
insu icien analy ical u ili y. Reducing o MIDI o
o he ixed-pi ch ep esen a ions o e ly simpli ies he
musical con en , he eby losing c i ical s ylis ic nuances.
Addi ionally, ajec o ies can se e as e ec i e okens
o aining AI sys ems, cap u ing long- ange s ylis ic
con inui y and s uc u al cohe ence mo e e ec i ely
han di ec sample-by-sample audio syn hesis app oaches,
which ypically s uggle o main ain long- ange musical
consis ency [5].
2. RELATED WORK
Recen decades ha e seen musicology, e hnomusicology,
and music in o ma ion esea ch (MIR) acknowledge he
limi s o Wes e n s a no a ion o o al and non-Wes e n
adi ions. The IDTAP builds on his by o e ing an
in e ac i e pla o m ha combines idioma ic ansc ip ion,
de ailed analy ic capabili y, and co pus-building. By
con as , p io wo k alls in o i e ca ego ies: (1)
au oma ed MIR pipelines ocused on ea u e ex ac ion
and one-sho analysis, wi h no mechanism o building
human-in o med, eusable co po a; (2) open-access
bu close-ended da ase s and a chi es; (3) no a ion
o ep esen a ion amewo ks—ei he idiom-speci ic o
“uni e sal”— ha a e no in eg a ed in o in e ac i e
analysis pla o ms; (4) educa ional and isualiza ion
en i onmen s ha en ich concep ual unde s anding bu
do no le use s add eco dings o au ho ansc ip ions
o expand a chi al co po a; and (5) anno a ion so wa e
ha suppo s audio analysis bu lacks idiom-speci ic
ansc ip ion o open-access a chi ing.
2.1 Sou h-Asian MIR
A subs an ial body o esea ch has explo ed compu a ional
app oaches o Sou h-Asian a -music co po a,
including esea ch on pi ch in ona ion [6–11], o ms
o con inuous-pi ch con ou s [12–18], and melodic
pa e n anno a ion and analysis [19–27]. These s udies
emphasize au oma ed ea u e ex ac ion and one-sho
analyses. The IDTAP ex ends his wo k by enabling
manual, idioma ically in o med ansc ip ion and i e a i e
analysis, suppo ing he c ea ion o eusable co po a and
compa a i e s udy.
2.2 Sou h-Asian A chi es and Da ase s
Music in Mo ion is a web a chi e ha con ains ideos ha
display melodic con ou plo s wi h ex ual anno a ions o
ly ical con en sc olling in eal ime wi h audio playback
[28]. Resea che s a Du ham Uni e si y ha e published
a da ase o hy hmic e en s ha is openly a ailable o
esea che s o use in iming- and hy hm- ocused empi ical
s udies [29]. The Music Technology G oup (MTG) a
Pompeu Fab a Uni e si y main ains he Sa aga collec ions,
he la ges anno a ed open da a collec ions a ailable
o compu a ional esea ch on Indian A Music [30].
Resea che s a Geo gia Tech ha e published he Sanidha
da ase , he i s open-sou ce Ca na ic music da ase
ea u ing S udio-quali y, mul i- ack audio eco dings wi h
high-de ini ion ideos o indi idual a is s’ pe o mances
[31]. These a chi es and esou ces a e, howe e ,
close-ended: hey do no p o ide an in e ac i e esea ch
en i onmen , as he IDTAP does.
2.3 "Uni e sal" and Non-Wes e n Music
Rep esen a ions
One ecen cus om no a ion sys em which has in luenced
he design o he IDTAP’s g aphical ep esen a ion is
And ew Killick’s "Global No a ion" [32]. Killick’s sys em
o e s inno a i e isual s a egies aimed a suppo ing
c oss-cul u al and compa a i e music analysis, aspi ing
owa ds a global, gene al amewo k o ep esen ing
di e se musical adi ions. While he IDTAP has c oss
cul u al applicabili y—pa icula ly o melisma ic- ich
adi ions—i is no s uc u ed o global ep esen a ion.
Ra he , i is speci ically ailo ed o idioma ic p ecision
and in e ac i e compu a ional esea ch ia so wa e-based
implemen a ion.
Simila ly, p ojec s encoding Chôngganbo [33], Tu kish
makam [34], o Ca na ic Sa gam [35] p o ide aluable
amewo ks o no a ing adi ion-speci ic ea u es, bu
wi hou in eg a ing hose no a ions in o an in e ac i e,
analy ical so wa e pla o m as IDTAP does.
2.4 C oss-Cul u al Compu a ional Musicology
MTG’s CompMusic [36], Dunya [37], and Musical
B idges [38] ha e signi ican ly b oadened MIR o
include di e se wo ld music adi ions, o e ing
in e ac i e lis ening and concep ual explo a ion ools.
IDTAP complemen s his wo k by p o iding in e ac i e,
idiom-awa e ansc ip ion and analysis ools ha di ec ly
gene a e eusable, compu a ionally legible da a o
c oss-cul u al and compa a i e s udy.
2.5 Musicological Analysis So wa e
Sonic Visualize , an open-sou ce desk op applica ion,
enables esea che s o inspec , anno a e, and analyze
audio h ough high- esolu ion wa e o ms, spec og ams,
pi ch and hy hm displays, and ex ensible plugin ea u e
laye s [39]. Unlike such gene al-pu pose ools, he
IDTAP in eg a es idiom-speci ic ansc ip ion, s uc u al
anno a ion, and co pus-building in a uni ied pla o m.
P oceedings o he 26 h ISMIR Con e ence, Daejeon, Ko ea, Sep embe 21-25, 2025
408
3. DISCIPLINARY INTERVENTIONS
The undamen al p inciple o he IDTAP is o di e si y
music esea ch, ep esen a ion, c ea ion, c i icism, and
heo y h ough mo e expansi e ansc ip ion o ms and
co esponding analy ics. This p inciple e lec s a
commi men o empi ical and compa a i e esea ch ha is
independen o 19 h-cen u y colonial axonomies, which
posi ion Eu opean acial and cul u al measu es as he
apex a ound which o he human and cul u al c ea ions a e
assessed and judged. We belie e he IDTAP, along wi h i s
esul an no a ions and da ase s, will be aluable o a wide
ange o esea ch/schola ship and eaching/lea ning.
Fo one, he IDTAP is designed o di e si y humanis ic
inqui y and suppo he p o essionaliza ion o he nex
gene a ion o esea che s. The oles o music ansc ip ion
and he empi ical analysis o music, once dis inc i e
me hodological ea u es in he ields o musicology and
e hnomusicology, a e cu en ly in lux. The musicological
u n in U.S. music depa men s in he 1980s and ’90s
decen e ed he ole o music heo y and o he empi ical
app oaches in a o o c i ical and cul u al heo e ical
app oaches [40–43]. This shi occu ed pa ly because
music ansc ip ion and analysis we e elian on s a
no a ion, e en when he music being s udied did no
align wi h he Eu opean classical adi ion o which s a
no a ion was o iginally de eloped.
The e ha e been wo b oad co ec i es o his si ua ion:
he abandonmen o ansc ip ion and empi ical me hods
al oge he , and he c ea ion o cus om no a ion sys ems.
Bo h solu ions, howe e , aise addi ional challenges. The
u n away om ansc ip ion-based esea ch mo es he
domain o melodic/ hy hmic sound analysis away om
quan i a i ely measu able e idence-based esea ch, as well
as cu ing edge compu a ional echnologies based on
digi al audio esea ch. This shi emo es a cen al pa
o e e yday li e om humanis ic and social scien i ic
inqui y: he in e connec ed ela ionship be ween melodic
adi ions and hei emo ional and a ec i e communi ies
[44]. The second co ec i e, cus om ansc ip ion me hods,
lack commensu a e ields o da a o quan i a i e and/o
compu a ional analysis, le alone a common language o
schola s o engage a b oade communi y o eade s and
esea che s.
The IDTAP is designed, hen, as a esea ch ool
o schola s o o al melodic adi ions and exp essi e
cul u es. S a is ically o ien ed esea che s can use
he da a ha unde gi ds his isual ep esen a ion o
ask new, quan i a i e kinds o esea ch ques ions.
E hnomusicologis s can use “big da a” compu a ional
app oaches o que y his a chi e a a la ge scale
ac oss a ange o pe o mances (diach onically o
synch onically), o a a small scale h ough speci ic
measu emen s wi hin a single pe o mance, o e en a
single ph ase. Music heo is s can use he da a as a
pla o m o he disco e y/cons uc ion o new kinds o
heo e ical knowledge ega ding melodic con ou , mac o-
and mic o- hy hm, and pe o me s yle. Social his o ians,
olklo is s, eligious s udies schola s, and musicologis s
can supplemen hei esea ch o (a ec i e) communi ies,
classes, cas es, o indi iduals h ough he ea men o
eco ded melodic sound as his o ical e idence.
The IDTAP also se es as a pedagogical ool
o unde g adua e and g adua e s uden s, as well as
independen musicians and music s uden s who wan o
each hemsel es. Whe eas gene a ions o p ac i ione s
ha e adi ionally lea ned unde he di ec u elage
o he edi a y o discipula o ebea s, he IDTAP has
he po en ial o os e a mo e di e se communi y o
p ac i ione s, including hose who we en’ bo n in o a
musical amily and who may no ha e p i ileged access o
a mas e - eache , due o socio-cul u al, economic, and/o
geog aphical ac o s.
Finally, he IDTAP se es as a co ec i e o he
bias cu en ly p esen in MIR, Music AI esea ch, and
co pus s udies, which ha e adi ionally ocused on
Wes e n musical co po a [45]. These epis emological
and da a-bo ne biases may ha e a - eaching consequences
o e he coming decades, as AI sys ems play an inc easing
ole in he p ocesses by which music is gene a ed,
dissemina ed, and consumed. To include non-Wes e n and
non-no a ed musical knowledge, hei unique o ms and
s uc u es mus i s be made compu a ionally legible.
4. IDTAP FRAMEWORK AND THEORETICAL
FOUNDATION
The IDTAP has been designed, de eloped, and
implemen ed wi h he ollowing componen s: 1) an
a chi e o eco dings, including ools o use s o
upload, documen , and lis en o he eco dings; 2) an
edi o in e ace, which includes a sui e o ools o use s o
compose o ansc ibe mul i- ack ansc ip ions consis ing
o melodic con ou s and ins umen al, syllabic, and/o
ex ual a icula ions, as well as o segmen and anno a e he
ansc ip ions; 3) an a chi e o ansc ip ions/no a ions;
4) da a- isualiza ion and da a-que y unc ions; and
5) collec ions o ganized by use unc ion: esea ch,
pedagogy, app ecia ion, and c ea i e p oduc ion. The
IDTAP use in e ace and analysis sui e enable and make
a ailable digi al ansc ip ions o schola s and s uden s
who ha e a chi al, language, linguis ic, musical, o
li e a y esea ch skills—common in A s, Humani ies and
Social Science disciplines—bu who may no be ained in
p og amming o compu a ional me hodologies.
The IDTAP amewo k is o ganized a ound wo co e
concep s: ajec o ies (see abo e) and a icula ions.
A icula ions coincide wi h momen s o melodic
change—such as no e onse s, o se s, o ins an aneous
pi ch ansi ions—and di e acco ding o he ins umen
being ansc ibed. Vocal a icula ions use In e na ional
Phone ic Alphabe (IPA) [46] symbols, allowing use s
o anno a e each ajec o y wi h a s a ing consonan ,
owel, and ending consonan , while ins umen al
a icula ions dis inguish playing echniques such as
plucking, bowing, blowing, p essing, o hamme ing.
Each ins umen , ocal o non- ocal, has i s own se
o a ailable ajec o ies and a icula ions, e lec ing i s
P oceedings o he 26 h ISMIR Con e ence, Daejeon, Ko ea, Sep embe 21-25, 2025
409
unique exp essi e cha ac e is ics.
Figu e 1. The hi een ajec o ies used in si a (plucked
s ing) ansc ip ions and he eigh ajec o ies used in
ocal and sa angi (bowed s ing) ansc ip ions.
5. IDTAP EDITOR INTERFACE
The “Edi o ” in e ace allows use s o ansc ibe a speci ic
eco ding—o make no a ions om sc a ch—by inse ing
ajec o ies on o a wo-dimensional ime-by-pi ch g aph.
To ansc ibe, use s place o ien a ion poin s on he
g aph wi h an o e laid spec og am and/o pi ch ace
and selec om an alphabe o ajec o ies—see Figu e
1— o ind he cu e ha mos closely i s he melodic
con ou . Fo ce ain ajec o ies, he slope o he cu e
can be adjus ed o be e ma ch he melodic con ou being
ansc ibed. O ien a ion poin s can also be “o se ” o
ep esen mic o ones (sh u is)—o spaces be ween scale
ones—and melodic con ou s ha mo e be ween scale
ones and mic o ones (andolans). Figu e 2 illus a es
wo ansc ip ions o a sec ion o he Hindus ani hum i
composi ion, “Babul Mo a,” pe o med by Begum Akh a .
As shown, he IDTAP is highly a uned o he nuances o
mo emen s bo h be ween and on scale ones.
Figu e 2. T ansc ip ion o he same selec ion o he song,
Babul Mo a, using s a and IDTAP no a ions.
5.1 G aphical Display
The IDTAP edi o in e ace ep esen s pi ch on a g aph
using a loga i hmic equency scale wi h adjus able uppe
and lowe bounds. Use s can cus omize he isual
elemen s ex ensi ely, including changing colo s o cla i y
and accommoda ion. Ho izon al lines indica e he p ima y
scale deg ees (swa a) o he cu en scale sys em ( aga),
and hei p ecise mic o onal uning can be adjus ed
o ma ch he speci ic pi ch con igu a ions o indi idual
pe o mances o adi ions.
The g aphical display suppo s up o ou ins umen al
o ocal acks simul aneously, each p esen ed in dis inc
use -selec ed colo s. Va ious ins umen al a icula ions a e
ep esen ed h ough symbolic no a ion: o si a , s anda d
plucks a e displayed as iangles, chika i s ing plucks
as “X" symbols, and s ing mu es as e ical lines a
he endpoin s o ajec o ies; o ocal acks, consonan s
appea as diamond shapes.
Addi ionally, use s can oggle isual o e lays, including
a spec og am [47] and a pi ch ace [48]. The spec og am
isualiza ion is adjus able in bo h in ensi y and colo map,
enhancing i s analy ical u ili y. These g aphical ools
enable use s o isually e i y ansc ip ion accu acy,
clea ly disce n nuanced melodic beha io s, and e ec i ely
compa e mul iple pe o mance laye s.
5.2 Me e
The IDTAP in e ace suppo s lexible speci ica ion,
isualiza ion, and adjus men o hie a chically nes ed
me ical s uc u es (e.g. me e o aal). Use s can
de ine me ical empla es and place hem p ecisely a
desi ed momen s wi hin a ansc ip ion. The pla o m
allows ine- uning o mic o- hy hmic sub-di ision
placemen o align closely wi h he na u al empo al
luc ua ions ( uba o) inhe en in pe o mance p ac ice.
Addi ionally, use s can adjus empo dynamically, se
a ying subdi isions wi hin each hie a chical laye ,
and isually inse alignmen poin s di ec ly on o he
ansc ip ion. An in eg a ed algo i hm au oma ically
calcula es he op imal me ical alignmen , de e mining
he app op ia e empo and bea placemen based on hese
use -de ined e e ence poin s.
5.3 Tuning
The IDTAP in e ace includes lexible uning ools ha
accommoda e he p ecise mic o onal equi emen s o
di e se musical adi ions. Use s can globally adjus
he undamen al equency o he ansc ip ion’s onic
pi ch (“sa”), au oma ically shi ing he e ical alignmen
o associa ed isual elemen s, including ho izon al
pi ch- e e ence lines and spec og am bounda ies.
Addi ionally, each scale deg ee can be indi idually
ine- uned, enabling p ecise pi ch adjus men s in cen s
ela i e o equal empe amen . To acili a e accu a e
uning, sine- one playback o each pi ch can be ac i a ed,
allowing use s o au ally align hese e e ence ones wi h
he o iginal audio eco ding. Simul aneously, isual
alignmen be ween adjus ed pi ch- e e ence lines and
co esponding melodic con ou s on he spec og am
suppo s use s in e ining uning con igu a ions.
P oceedings o he 26 h ISMIR Con e ence, Daejeon, Ko ea, Sep embe 21-25, 2025
410
5.4 Segmen a ion and Anno a ion
The IDTAP in e ace p o ides obus ools o segmen ing
ansc ip ions a mul iple hie a chical le els, enabling
p ecise s uc u al ep esen a ion and de ailed analy ical
anno a ion. T ansc ip ions can be di ided e ically in o
wo dis inc hie a chical laye s: Sec ions and Ph ases.
Sec ion di isions ep esen b oade s uc u al elemen s
(e.g., composi ions o imp o isa ional segmen s), wi h
hick e ical lines, whe eas Ph ase di isions ep esen
ine subdi isions wi hin Sec ions, ma ked by hinne
e ical lines. Use s can inse and adjus hese di isions
in ui i ely, using simple mouse ac ions o keyboa d
sho cu s.
Anno a ions o each segmen o ph ase can be applied
using he Label Edi o , allowing o de ailed desc ip i e
analysis. Fo Sec ions, use s may label b oade ca ego ies
such as ‘Imp o isa ion,’ ‘Composi ion,’ ‘Alap,’ o speci y
a pa icula ype o composi ion. Ph ase-le el anno a ion
o e s e en g ea e g anula i y, enabling use s o selec
om mul iple desc ip o s ac oss se e al dimensions:
Ph ase-Types (composi ional o ms); Elabo a ion-Types
(imp o isa ional o ms); A icula ion-Types (e.g., i a
ocalis is singing ly ics (bols)), No e-Names (sa gam),
Non-Lexical syllables (aaka ,nom- om); and Inciden als
(e.g., uning, alk, o con e sa ions by musicians).
Mul iple labels can be concu en ly applied o any single
ph ase, e lec ing he complexi y o pe o mance p ac ice.
5.5 Syn hesis
The IDTAP in eg a es a eal- ime syn hesis module o
acili a e ansc ip ion. Implemen ed wi h he Web Audio
API, his module ende s he cu en ansc ip ion as
audio, allowing use s o audi ion he syn hesized endi ion
alongside he o iginal eco ding in o de o i e a i ely
e ine a ansc ip ion’s accu acy.
Fo ocal ansc ip ions, he IDTAP employs he
Kla Syn hesize , a o man syn hesis model designed
o simula e human speech and ocal imb es [49].
Plucked-s ing ins umen s, such as he si a , a e
syn hesized using he Ka plus-S ong algo i hm, a physical
modeling echnique e ec i e o simula ing plucked s ing
acous ics [50]. Bowed s ing ins umen s, such as he
sa angi, a e syn hesized using a cus om-designed engine
ha inco po a es dynamically modula ed eedback loops,
esonan il e ing, and noise-based exci a ion o emula e
acous ic in e ac ions ypical o bowed cho dophones.
In ansc ip ions in ol ing mul iple ins umen s, each
ins umen is assigned a dedica ed syn hesize . A
mixe in e ace enables use s o independen ly adjus he
olume le els o he o iginal eco ding and syn hesized
acks, acili a ing p ecise audi o y compa isons sui ed
o de ailed ansc ip ion and analysis asks. Addi ional
ea u es include looping o speci ic ansc ip ion segmen s,
adjus able playback speeds, and pi ch shi ing capabili ies
ha al e he undamen al equency o bo h syn hesized
audio and o iginal eco dings. These unc ionali ies
suppo ansc ip ion wo k lows ha equi e alignmen
wi h ins umen s uned o di e en e e ence onics.
6. ANALYSIS, DATA VISUALIZATION, AND
QUERY PROCESS
The “Analysis” in e ace includes in e ac i e ools o
use s o engage wi h ansc ip ion da a heu is ically.
These ools include: 1) a que y sys em ha isola es
and displays speci ic sequences om a ansc ip ion
based on an unlimi ed numbe o il e ing c i e ia; 2) a
pi ch occu ence da a- isualiza ion ool ha displays he
ela i e p opo ion o ime spen on pa icula pi ches,
and segmen ed by sec ion, ph ase, o du a ional window;
3) a “pi ch pa e n” isualiza ion ool ha displays he
ela i e equencies o speci ic pi ch pa e ns h oughou
a ansc ip ion. This in e ace suppo s bo h emic and e ic
analyses: Emic app oaches allow analys s o anno a e and
analyze based on “inside ” sensibili ies, ca ego ies, and
concep s, while e ic app oaches enable analysis based on
“neu al” pa ame e s [51].
6.1 Pi ch P e alence
The pi ch p e alence in e ace allows use s o gene a e
isualiza ions ep esen ing he p e alence o pa icula
pi ches o e he cou se o a ansc ip ion. This is achie ed
by spli ing he ansc ip ion in o du a ional windows
acco ding o one o h ee use -selec ed segmen a ion
schemes. One segmen a ion scheme—by du a ion—is
e ic, as i di ides he ansc ip ion in o windows o
use -de ined du a ion. This me hod aims o o e come
a ious biases: bo h inside knowledge o adi ional
concep ions o ph ase s uc u e as well as ou side in ui i e
pe cep ions o ph ase s uc u e. Two o he segmen a ion
schemes—by sec ion and by ph ase—a e emic app oaches,
elying on an analys ’s inside knowledge and hei
decisions on whe e o place analy ical in en ions in
explici ly en e ing ph ase and sec ion di isions h oughou
he p ocess o ansc ip ion. Fo he emic segmen a ion
app oaches, he use is also able o il e he segmen s o
include o exclude pa icula sec ion o ph ase ypes, as
anno a ed by he use du ing ansc ip ion. Wi h he sec ion
segmen a ion, a use may choose o include o exclude
sec ion ypes: in he case o Hindus ani music, di e en
o ms o composi ion and imp o isa ion.
Fo example, Figu e 3 displays he pi ch p e alence and
du a ion o he i s six sec ions and ele en ph ases o he
hum i song, “Babul Mo a,” pe o med by Begum Akh a ,
along wi h an anno a ed desc ip ion o each sec ion and
ph ase ype, allowing use s o see a a glance bo h he
o mal s uc u e and pi ch dis ibu ion o he pe o mance.
In addi ion o he ype o segmen a ion, use s can oggle
be ween: a) a isualiza ion o abula e he p e alence
o each pi ch ch oma (i.e. pi ch class, he quali y o a
pi ch as abs ac ed om i s oc a e egis a ion) ins ead o
indi idual pi ches; b) a e ically condensed display, which
excludes emp y e ical space o pi ches no included in
he ansc ip ion; c) a “hea map” g ayscale g adien ha
ep esen s he ela i e p e alence o elemen s, a he han
showing he mos common (mode) elemen in da k g ay
and he es in ligh g ay.
P oceedings o he 26 h ISMIR Con e ence, Daejeon, Ko ea, Sep embe 21-25, 2025
411
Figu e 3. Pi ch P e alence Da a Visualiza ion.
6.2 Que y Display
The que y display in e ace allows use s o c ea e
mul i-pa que ies o il e a ansc ip ion, displaying only
hose segmen s ha ma ch speci ic c i e ia. Fo example,
Figu e 4 demons a es he que y unc ion and ou pu o
e e y ins ance when Begum Akh a sings “babul” ( a he ).
The que y, in his case, looks o a ph ase con aining
a s a ing consonan “ba,” a owel “u,” and an ending
consonan “l”. This ype o que y could be used o
pedagogical o analy ical in es iga ions in o how an a is
exp esses a composi ion—whe e wo d and melody a e
ixed—in mul iple ways, isually examining he in e play
be ween composi ion and imp o isa ion.
Figu e 4. Que y Func ions and display example.
Use s can cons uc que ies based on a wide ange
o ca ego ies, including: Pi ch, S ic Pi ch Sequence
(pi ches mus appea in his exac o de wi hou any
gaps be ween hem), Loose Pi ch Sequence (pi ches mus
appea in his o de , bu may ha e gaps be ween hem),
T ajec o y, S ic T ajec o y Sequence, Loose T ajec o y
Sequence, Sec ion Type, Alap Type, Composi ion Type,
Tempo, Tala (desc ip o o me ic s uc u e and hie a chy,
akin o a Time Signa u e in wes e n music), Ph ase Type,
Elabo a ion Type, Inciden al Type, o A icula ion Type.
Fo each ca ego y, use s can selec a designa o :
“includes,” “excludes,” and, i applicable, “s a s wi h,”
o “ends wi h.” Fo example, a que y migh include all
ph ases ha con ain he pi ch “Sa” o exclude all ph ases
ha con ain a speci ic ajec o y, such as a “k in in-slide.”
This que ying sys em is especially powe ul when use s
combine mul iple que ies, allowing hem o ocus on
speci ic analy ical ques ions o hypo heses. Use s can
choose o display hose segmen s ha sa is y all o he
que ies, o jus a leas one o he que ies. Once he desi ed
ph ases a e displayed, analys s can explo e he il e ed
esul s and d aw insigh s om hei obse a ions.
6.3 Pi ch Pa e ns
The Pi ch Pa e ns in e ace enables use s o selec a
speci ic ype o segmen a ion and pa e n size. I hen
coun s and eco ds he numbe o occu ences o each
unique pi ch sequence wi hin each segmen , based on
he chosen pa e n size, and displays his in o ma ion
alongside a g aphical depic ion o he con ou s o each
sequence. This kind o display allows analys s o
in ui i ely sea ch o common pa e ns o a ious sizes
in di e en segmen s h oughou a pe o mance, hus
e ealing some essence o he s yle o he melodic
elabo a ion.
6.4 Analysis Sui e Implica ions
Toge he , he ypes o analysis enabled by hese que ies,
da a isualiza ions, and da ase s a e mul iple. To name
jus a ew, use s can examine a pa icula eco ding,
sec ion om a eco ding, o body o eco dings associa ed
wi h an indi idual o g oup o musicians om a
ange o adi ions. Resea ch opics could include:
ela ionships be ween ly ic- ex and melody, o syllable
and melodic mo emen ; empi ically measu able changes
o a musician’s pe o mance s yle o e ime: h oughou
an indi idual pe o mance o ac oss many pe o mances;
analysis ac oss ins umen s, enues, s yles, pe o mance
sec ions, ime-pe iods, geog aphic a eas, eco ding media,
o o he clea ly speci ied me hod o slicing a co pus. These
h ee analysis ools will be enhanced and added o as
ou p ojec eam con inues o engage wi h and espond
o he needs o ou esea ch colleagues. The IDTAP
also le s use s expo da a in JSON o CSV, so hey can
conduc quan i a i e analyses in whiche e so wa e o
p og amming en i onmen hey p e e .
7. CONCLUSION
Exis ing musical ep esen a ion schemes ine i ably e lec
hei socio-his o ical con ex s, embedding assump ions
abou musical s uc u es, o ms, and meanings. The
IDTAP p o ides a new amewo k o ep esen and analyze
o al melodic adi ions p ecisely and lexibly, esponding
di ec ly o he nuanced demands o Hindus ani music. By
cen e ing exp essi e musical ea u es ypically o e looked
by adi ional no a ion sys ems, he pla o m aims o os e
new esea ch me hodologies, pedagogical oppo uni ies,
and c ea i e p ac ices. As compu a ional app oaches o
music schola ship g ow inc easingly signi ican , pla o ms
like IDTAP help ensu e ha di e se musical adi ions
emain isible, analyzable, and c ea i ely i al in he
digi al age.
P oceedings o he 26 h ISMIR Con e ence, Daejeon, Ko ea, Sep embe 21-25, 2025
412
8. ACKNOWLEDGMENTS
This wo k was suppo ed by he Na ional Endowmen o
he Humani ies (g an HAA-290356-23) and by se e al
p og ams a he Uni e si y o Cali o nia, San a C uz:
he A s Resea ch Ins i u e, he O ice o Resea ch, and
he Hasan and Ali Akba Khan Endowmen o Classical
Indian Music. We hank he ISMIR e iewe s o
p o iding aluable eedback on his esea ch. We dedica e
his pape o he memo y o ou colleague, men o , and
iend La y Polansky.
9. REFERENCES
[1] C. Seege , “Towa d a Uni e sal Music Sound-W i ing
o Musicology,” Jou nal o he In e na ional Folk
Music Council, ol. 9, pp. 63–66, Jan. 1957.
[2] N. M. England e al., “Symposium on T ansc ip ion
and Analysis: A Hukwe Song wi h Musical Bow,”
E hnomusicology, ol. 8, no. 3, pp. 223–233, Sep.
1964.
[3] M. Hood, The E hnomusicologis . New Yo k, 1971.
[4] T. Ellingson, “T ansc ip ion,” in E hnomusicology:
An In oduc ion, H. Mye s, Ed. Basings oke, 1992,
pp. 153–164.
[5] K. Goel e al., “I ’s Raw! Audio Gene a ion wi h
S a e-Space Models,” In e na ional Con e ence on
Machine Lea ning, PMLR, 2022.
[6] S. ¸Sen u k e al., “A Sco e-In o med Compu a ional
Desc ip ion o S a as Using a S a is ical Model,” in
P oceedings o he 13 h Sound and Music Compu ing
Con e ence, Hambu g, Ge many, 2016.
[7] G. k. Kodu i e al., “Cha ac e iza ion o In ona ion in
Ca na ic Music by Pa ame izing Pi ch His og ams,” in
ISMIR, Po o, Po ugal, 2012.
[8] J. Se à e al., “Assessing he Tuning o Sung Indian
Classical Music,” in ISMIR, 2011.
[9] M. M. Koma agi i, “An Empi ical S udy o In ona ion
in Raga Kalyani,” in P oceedings o he In e na ional
Symposium "IT- o-Mass: Speech and Music",
Bhubaneswa , India, 2005.
[10] K. Akan e al., “Measu ing F equencies o Sh u is
in Indian Classical Music,” in P oceedings o he
9 h In e na ional Con e ence on Eme ging T ends in
Enginee ing and Technology—Signal and In o ma ion
PRocessing, 2019.
[11] M. M. Koma agi i e al., Pi ch Analysis in Sou h Indian
Music: Wi h a C i ical Examina ion o he Theo y o 22
´
S u i-s. New Delhi: Munshi am Manoha lal, 2013.
[12] V. S. Vi a agha an, R. A a ind, and H. A. Mu hy, “A
S a is ical Analysis o Gamakas in Ca na ic Music,” in
P oc. ISMIR, Suzhou, China, Oc . 2017, pp. 243–249.
[13] R. Mo is, “Tana Va nam-s: An En y in o R¯
aga
Delinea ion in Ca na ic Music,” Analy ical App oaches
o Wo ld Music, ol. 1, no. 1, pp. 1–27, 2011.
[14] T. Nu all, X. Se a, and L. Pea son, “S a a-Fo ms
and Coa icula ion in Ca na ic Music: An In es iga ion
Using Deep Clus e ing,” in P oceedings o he
11 h In e na ional Con e ence on Digi al Lib a ies
o Musicology (DL M), S ellenbosch, Sou h A ica,
Jun. 2024.
[15] T. Nu all, X. Se a, and L. Pea son, “S a a-Fo ms
in Ca na ic Music: Con ex ual In luences on he
Pe o mance o S a a,” in Wo kshop on Indian Music
Analysis and Gene a i e Applica ions (WIMAGA),
sa elli e wo kshop o IEEE ICASSP, Hyde abad, India,
Ap . 2025.
[16] L. Pea son, “Coa icula ion and Ges u e: An
Analysis o Melodic Mo emen in Sou h Indian
Raga Pe o mance,” Music Analysis, ol. 35, no. 3,
pp. 280–313, 2016.
[17] M. Sub amanian, “Ca na ic R¯
agam Thodi – Pi ch
Analysis o No es and Gamakams,” Jou nal o he
Sangee Na ak Akademi, ol. 41, no. 1, pp. 3–28, 2007.
[18] M. Sub amanian, “Ca na ic Music – Au oma ic
Compu e Syn hesis o Gamakams,” Jou nal o he
Sangee Na ak Akademi, ol. 43, no. 3, pp. 28–36,
2009.
[19] T. Nu all e al., “In Sea ch o Sañca as:
T adi ion-In o med Repea ed Melodic Pa e n
Recogni ion in Ca na ic Music,” in P oceedings o
he 23 d In e na ional Socie y o Music In o ma ion
Re ie al Con e ence, Bengalu u, India, 2022.
[20] K. K. Ganguli e al., “Da a-D i en Explo a ion
o Melodic S uc u e in Hindus ani Music,” in
P oceedings o he 17 h In e na ional Socie y o
Music In o ma ion Re ie al Con e ence, New Yo k,
NY, USA, 2016, pp. 605–611.
[21] S. Gula i, Compu a ional App oaches o Melodic
Desc ip ion in Indian A Music Co po a. Ph.D.
disse a ion, Uni e si a Pompeu Fab a, Ba celona,
Spain, 2017.
[22] S. Gula i e al., “Mining Melodic Pa e ns in
La ge Audio Collec ions o Indian A Music,”
in P oceedings o he In e na ional Con e ence on
Signal Image Technology and In e ne Based Sys ems,
Ma akech, Mo occo, 2014.
[23] S. Gula i e al., “Disco e ing Raga Mo i s by
Cha ac e izing Communi ies in Ne wo ks o Melodic
Pa e ns,” in P oceedings o he IEEE In e na ional
Con e ence on Acous ics, Speech, and Signal
P ocessing, Shanghai, China, 2016, pp. 286–290.
P oceedings o he 26 h ISMIR Con e ence, Daejeon, Ko ea, Sep embe 21-25, 2025
413
[24] V. Ishwa e al., “Mo i Spo ing in an Alapana
in Ca na ic Music,” in P oceedings o he 14 h
In e na ional Socie y o Music In o ma ion Re ie al
Con e ence, Cu i iba, B azil, 2013, pp. 499–504.
[25] T. Nu all e al., “The Ma ix P o ile o Mo i
Disco e y in Audio – An Example Applica ion
in Ca na ic Music,” in P oceedings o he 15 h
In e na ional Symposium on Compu e Music
Mul idisciplina y Resea ch, Tokyo, Japan, 2022.
[26] P. Rao e al., “Melodic Mo i ic Analysis o Indian
Music,” Jou nal o New Music Resea ch, ol. 43, no. 1,
pp. 115–131, 2014.
[27] P. Rao e al., “Classi ica ion o Melodic Mo i s in Raga
Music wi h Time-Se ies Ma ching,” Jou nal o New
Music Resea ch, ol. 43, no. 1, pp. 115–131, 2014.
[28] S. Rao and W. an de Mee , “Music in Mo ion: The
au oma ed ansc ip ion o Indian music (AUTRIM)
p ojec by NCPA and U A,” 2017. [Online]. A ailable:
h ps://au imncpa.wo dp ess.com/.
[Accessed Jan. 16, 2024].
[29] M. Clay on, L. Lean e, and S. Ta si ani, “IEMP No h
Indian Raga,” Da abase o anno a ed audio isual
eco dings, 2018. [Online]. A ailable: h ps://
doi.o g/10.17605/OSF.IO/KS325.
[30] A. S ini asamu hy e al., “Sa aga: Open Da ase s o
Resea ch on Indian A Music,” Empi ical Musicology
Re iew, ol. 16, no. 1, pp. 85-98, 2021.
[31] V. V. K ishnan e al., “Sanidha: A S udio Quali y
Mul i-Modal Da ase o Ca na ic Music,” in ISMIR,
2024.
[32] A. Killick, “Global No a ion as a Tool o
C oss-Cul u al and Compa a i e Music Analysis”
Analy ical App oaches o Wo ld Music, ol. 8, no. 2,
pp. 235–279, 2020.
[33] J. Lee, J. Downie, and A. Renea , “Rep esen ing
Ko ean T adi ional Musical No a ion in XML,” in
ISMIR, 2002.
[34] M. Ka aosmano˘
glu, “A Tu kish Makam Music
Symbolic Da abase o Music In o ma ion e ei al:
SymbT ,” in ISMIR, 2012.
[35] S. Mammen, e al., “iSa gam: Music No a ion
Rep esen a ion o Indian Ca na ic Music,” EURASIP
Jou nal on Audio, Speech, and Music P ocessing,
ol. 2016, no. 5, 2016.
[36] X. Se a, “A Mul icul u al App oach in Music
In o ma ion Resea ch,” in ISMIR, pp. 151-156, 2011.
[37] A. Po e , M. So do, and X. Se a, “Dunya: A Sys em
o B owsing Audio Music Collec ions Exploi ing
Cul u al Con ex ,” in ISMIR, 2013.
[38] Uni e si a Pompeu Fab a, Music Technology
G oup, “Musical B idges – B idging cul u es
h ough music unde s anding,” 2021. [Online].
A ailable: h ps://www.up .edu/en/web/
musicalb idges/. [Accessed: 13-Jun-2025].
[39] C. Cannam, C. Landone, and M. Sandle , “Sonic
Visualise : An Open Sou ce Applica ion o Viewing,
Analysing, and Anno a ing Music Audio Files,” in
P oceedings o he ACM Mul imedia In e na ional
Con e ence, 2010.
[40] S. Mcla y, Feminine Endings: Music, Gende , and
Sexuali y. Minneapolis: Uni . Minn. P ess, 1991.
[41] J. Ke man, Con empla ing Music: Challenges o
Musicology. Camb idge, MA: Ha a d Uni e si y
P ess, 1985.
[42] L. Goeh , The Imagina y Museum o Musical Wo ks:
An Essay in he Philosophy o Music. Ox o d:
Cla endon P ess, 1992.
[43] M. Ci on, Gende and he Musical Canon.
Camb idge: Camb idge Uni e si y P ess, 1993.
[44] L. Gandhi, A ec i e Communi ies: An icolonial
Though , Fin-de-siècle Radicalism, and he Poli ics o
F iendship. Du ham: Duke Uni e si y P ess, 2006.
[45] P. E. Sa age, “An o e iew o c oss-cul u al music
co pus s udies,” in Ox o d Handbook o Music and
Co pus S udies, D. Shanahan, A. Bu goyne, and I.
Quinn, Eds. New Yo k: Ox o d Uni e si y P ess, [In
p ess]. [Online]. A ailable: h p://doi.o g/10.
31235/os .io/nx bg.
[46] In e na ional Phone ic Associa ion, Handbook o he
In e na ional Phone ic Associa ion: A Guide o
he Use o he In e na ional Phone ic Alphabe .
Camb idge: Camb idge Uni e si y P ess, 1999.
[47] C. Schö khube , A. Klapu i, N. Holighaus, and
M. Dö le , “A MATLAB oolbox o e icien
pe ec econs uc ion ime- equency ans o ms wi h
log- equency esolu ion,” in P oc. 53 d In e na ional
Con e ence: Seman ic Audio, Audio Enginee ing
Socie y, Jan. 2014.
[48] J. Salamon and E. Gómez, “Melody ex ac ion
om polyphonic music signals using pi ch con ou
cha ac e is ics,” IEEE T ansac ions on Audio,
Speech, and Language P ocessing, ol. 20, no. 6,
pp. 1759–1770, 2012.
[49] D. H. Kla , “So wa e o a cascade/pa allel o man
syn hesize ,” Jou nal o he Acous ical Socie y o
Ame ica, ol. 67, no. 3, pp. 971–995, 1980.
[50] K. Ka plus and A. S ong, “Digi al syn hesis o
plucked-s ing and d um imb es,” Compu e Music
Jou nal, ol. 7, no. 2, pp. 43–55, 1983.
P oceedings o he 26 h ISMIR Con e ence, Daejeon, Ko ea, Sep embe 21-25, 2025
414
[51] M. Ha is, “His o y and Signi icance o he Emic/E ic
Dis inc ion,” Annual Re iew o An h opology, ol. 5,
pp. 329–350, 1976.
P oceedings o he 26 h ISMIR Con e ence, Daejeon, Ko ea, Sep embe 21-25, 2025
415
