Melody-rendering Method Based on Generative Theory of Tonal Music

Melody-rendering Method Based on Generative Theory of Tonal Music Masatoshi Hamanaka1,KeijiHirata 2,andSatoshiTojo 3 1RIKEN 2MIRAI SHARE 3Asia University Abstract. This paper presents a melody-rendering method that is based on the generative theory of tonal music (GTTM). A time-span tree can be obtained by analyzing a melody on the basis of GTTM. Our method is an inverse process of GTTM analysis and outputs a corresponding melody when a time-span tree is input. The melody-morphing method is a melody-generation method that is based on GTTM. With this method, it is possible to generate a new variation of melody by calculating the time-span tree of two melodies. However, melody morphing requires the difficult task of preparing another melody similar to the time-span tree of one melody. Our proposed melody-rendering method, however, can directly generate another melody from the time-span tree of one melody. We conducted an experiment to determine the effectiveness of our method, and the results indicate that all music pieces of 30 time-span trees in the GTTM database can be rendered with our method. Keywords: Melody Rendering ·Generative theory of tonal music (GTTM) ·Time-span tree. 1Introduction We propose a melody-rendering method for generating melodies corresponding to the time-span tree of the generative theory of tonal music (GTTM)[12]. The GTTM was proposed by Leardahl and Jackendoffin 1987, and a time-span tree is a binary tree with each branch connected to each note. The branches of a timespan tree are connected closer to the root than those connected to structurally important notes. Many attempts have been made to implement time-span-tree analysis on computers, but there are many analysis errors [8, 6,1]. There is a method of using deep learning to achieve high accuracy [2], and our rendering method is similar to this method in that it introduces an automatic translation framework All rights remain with the authors under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Proc. of the 17th Int. Symposium on Computer Music Multidisciplinary Research, London, United Kingdom, 2025 Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 961 M. Hamanaka, K. Hirata and S. Tojo into the analysis. However, when automatic translation is used as the analysis engine, an unexpected translation result may be output, and in such a case, the analysis process is interrupted in the middle. When encoding the training data, we explicitly indicate which note was reduced to which note, so that unexpected results will be less likely to occur. The main advantage of time-span trees is the ability to reduce notes. The time-span tree in Fig. 1 is the result of analyzing a melody (a) on the basis of GTTM. Reduced melodies can be extracted by cutting this time-span tree with a horizontal line and omitting the notes connected below the line (Figs. 1 (b)(c)). Melody reduction with GTTM is the absorption of notes by structurally important notes. Time-span tree (a) (b) (c) (b) (c) Fig. 1. Time-span tree and melody reduction Amelody-morphingmethodhasbeenproposedthatappliesthisreduction operation (Fig. 2) to generate a melody that is structurally intermediate between two input melodies by combining two melodies after executing reduction on the time-span trees of these two melodies [8,3,4]. However, combining two melodies cannot be executed with this method unless the structures of the time-span trees of the two melodies are similar (Fig. 2(c)). It is not easy to search for melodies with similar time-span-tree structures. In contrast, our method directly generates melodies with similar structures by rendering (Fig. 3) A melody corresponding to a time-span tree is generated while sequentially adding branches from the root. Our method has the following four features. Stepwise rendering: It is difficult to immediately obtain the melody corresponding to the time-span tree. Therefore, our method uses step-wise rendering, which repeats the rendering of one note. Branch priority: To create a training data set for stepwise rendering, it is necessary to define the branch priority. We use the maximum time-span tree defined on the basis of GTTM as the branch priority [7]. Encoding: By encoding the score into text, stepwise rendering can be learned in the framework of automatic translation. This makes it possible to add a Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 962 Melody-rendering Method Based on GTTM Melody A Melody B σA (a) Linking common events (b) Partial melody reduction (c) Combining two melodies Melody α Melody β Melody C σB σAσB σα i σβ j σασβ ︙ Fig. 2. Melody-morphing method note to a specified position in a melody as if adding a word to a specified position in a sentence. Time-span-tree matrix: The time-span tree has been handled in XML and Json formats, making coding difficult [5]. We made coding easier by expressing the information necessary for rendering, namely, pitch, note value, time-span-tree shape, and branch priority, in a matrix. We conducted an experiment to determine the effectiveness of our method, and the results indicate that all songs of 30 time-span trees in the GTTM database can be rendered with our method. The remainder of the paper is as follows. Section 2 prsents our melody-rendering method, and Section 3 describes its implementation. Section 4 describes the experimental results, and Section 5 gives a summary and mentions future plans. Time-span-tree analysis Melody rendering Fig. 3. Time-span-tree analysis and melody rendering Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 963 M. Hamanaka, K. Hirata and S. Tojo 2ProposedMelody-renderingMethod In our research, we are attempting to learn rendering, which is the inverse process of reduction, by using the Transformer model [16], an automatic translation network. There are 300 classical melodies of time-span-tree ground-truth data sets in the GTTM database[5]. These three hundred melodies are too small to directly learn from the score to a time-span tree with the Transformer model. With asmallnumberofmusicpiecesforlearningdata,over-fittingisinevitable,and an appropriate value cannot be output when unknown data are input. Therefore, our method uses stepwise reduction in which the minimum process of analysis is set as one data set; therefore, the number of data sets is increased. For example, if the deep neural net directly learns the relationship between a four-note melody and its time-span tree, the number of data sets is only one. If we consider the process of reducing one note to one data set, the number of data sets will be three. A time-span tree for a melody consisting of four notes can be constructed by estimating four to three notes, three to two notes, and two to one note, and combining the results (Fig. 4). 2.1 Branch Priority Time-span reduction removes decorative notes by pruning from the leaves at the tip of the tree, leaving only structurally important notes in the melody. To implement stepwise rendering, the branch priority must be obtained in a total order. However, there are only a few examples of reduction using the time-span tree regarding GTTM, and there is no detailed explanation on the reduction procedure [12]. For example, in Fig. 1, we can see two levels of reduction results, but it is not clear how many levels are necessary. Marsden et al.[13]suggestedawaytodeterminethesalienceoftwonote events a and b, neither of which are descendants of the other. They proposed defining the salience of an event as the duration of the maximum time spans of the two children at the branching point when the event is generated or where it is reduced. It is more important for the Transformer model to learn the relationship before and after rendering than it is to reduce the order of the notes to close to that of human cognition. In this subsection, we describe a time-span tree leveled by the duration of the time span for a simple reduction order that it is easy for the Transformer model to learn. The head in a time-span tree is the top-most pitch event, that is, the most salient in the tree. When two adjacent subtrees are combined, one of the two heads of the subtrees becomes the head of the whole. This indicates that the head of a tree is most salient in the time interval the tree occupies. Since a tree is a hierarchical combination of subtrees, the longest interval of each event in the tree is the most salient as the head of a subtree. Accordingly, we define the base case, when a subtree consists of a single pitch event, to be the duration of the event. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 964 Melody-rendering Method Based on GTTM Maximum time span: We call the longest temporal interval when a given pitch event becomes most salient as the maximum time span for the event. In other words, the maximum time span of a pitch event coincides with the temporal duration of the subtree of which the event becomes the head as a result of the time-span analysis. The priority of each branch of the time-span tree is determined with a timespan tree drawn with the maximum time span used in the time-span segmentation executed as the first step of the analysis of the time-span reduction. The branch priority is determined in accordance with the following rules. –Priorities are assigned to each level from the top of the time-span tree drawn with the duration of the time span. –At the top level, the main branches take precedence. –At the second and subsequent levels, the higher the priority of a branch X is, the higher the priority of the branch offof X becomes. The maximum time-span of the terminal branch is the length of the pitch event connected to the branch. The maximum time-span of a pitch event coincides with the temporal duration of the subtree of which the event becomes the head as a result of the time-span analysis (Fig. 5) [9]. qqqq qqh Data set 1 Data set 2 Data set 3 qqh h h h h w Fig. 4. Stepwise reduction Time-span tree Maximum time span Head Fig. 5. Maximum time-span tree Figure 6 shows a time-span tree drawn with the duration of the time span. The branch priority is determined in order from the top in accordance with the first rule. Then, in accordance with the second rule, branch 1 has the highest priority in this time-span tree, and branch 2 has the second-highest priority. The second level in this tree is the double-note level. In accordance with the second rule, the branch from 1 becomes 3, and that from 2 becomes 4. In the same way, the priority is determined up to the 16th note level. The same process applies to triplets, where the maximum length of unprocessed notes becomes the next level. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 965 M. Hamanaka, K. Hirata and S. Tojo q h e x w w w ) &##c.j.j.j １ １ １ １ 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 7 8 8 9 9 9 10 10 10 11 12 12 12 13 14 15 16 17 17 18 19 19 20 20 21 21 22 22 13 13 23 23 24 2425 26 26 27 28 29 30 31 32 33 3435 363738 39 40 41 Fig. 6. Time-span tree leveled by duration of time span 2.2 Learning and Evaluation Data The preparation of the data set for stepwise rendering is as follows. First, the priority of each branch of the time-span tree is evaluated on the basis of the duration of the maximum time span [2]. We call the longest temporal interval when a given pitch event becomes most salient as the maximum time span for the event. Next, stepwise reduction is applied to the least important note. A learning data set of stepwise rendering is then created using the data before stepwise reduction as output data and the data after reduction as input data (Fig. 7) Data set 1 Data set 2 Data set 3 qqqqqqhh h qqh q Rendering h hq Rendering wh Rendering Fig. 7. Stepwise rendering As a result of preparing the data sets, 7362 stepwise rendering training data sets are generated from 270 music pieces from the GTTM database consisting of Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 966 Melody-rendering Method Based on GTTM 300 pieces and 849 stepwise rendering evaluation data sets are generated from the remaining 30 pieces for evaluation. 2.3 Encoding Learning data are created from MusicXML and time-spanXML in the GTTM database. Since all melodies in the GTTM database are monophonic, the rendering method is limited to monophony. The notes in the melodies are made into a one-character string with the pitch and duration concatenated. The pitch is represented as 12 types without distinguishing between different octaves. By multiplying by 4, the duration of most notes becomes an integer, but since there are melodies containing only a few triplets, quintuplets, sextuplets, and septuplets, the duration is rounded up to an integer. The placeholders "l" or "r" are inserted at positions where notes disappeared due to the reduction. The "l" (left) is inserted when the reduction is absorbed into the left note, and "r" (right) is inserted when it is absorbed into the right note. Figure 8 is an example of learning data. Before rendering. → After rendering. c14 c16 d30 c26 l c16 d20 c16. → c14 c16 d30 c14 c12 c16 d20 c16 . c14 c16 d30 c26 r d36 c16. → c14 c16 d30 c26 c16 d20 c16 . c14 c16 d30 r d62 c16. → c14 c16 d30 c26 d36 c16 . c30 l d30 R2 d62 c16. → c14 c16 d30 d62 c16 . c60 l d62 c16. → c30 d30 d62 c16 . c62 r c78. → c60 d62 c16 . r c138. → c60 c78 . Fig. 8. Learning data for melody rendering 2.4 Data Augmentation The 7362 training data sets is not enough to train the Transformer model, so we carry out data augmentation. Each note is shifted 12 times by a semitone and the amount of training data is augmented by 12 times . The durations of notes are 2–16 times and rounded up to the the nearest integer, then the amount of data is augmented by 16 times . Finally, we prepare 1,432,704 (= 7362 x 12 x 16) learning data sets. 3 Implementation Amelodycorrespondingtothetime-spantreeisgeneratedbyrepeatingstepwise rendering. Representing a time-span tree as a matrix makes it easier to implement melody reduction and rendering in a program. In Fig 9, the first row of the matrix is the encoded pitch and duration and the second row is the connected parent branch number. The root branch has no parent branch to connect to, Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 967 M. Hamanaka, K. Hirata and S. Tojo so the parent branch number is set to 0. Both the 2nd and 4th branches are connected to the 1st branch, but the branches of the time-span tree do not cross [12], indicating that the 4th is connected to the 1st at a position closer to the root. Notes that are missing due to simplification or that have not been added due to rendering have blank pitches and durations on the matrix. The 3rd row of the matrix is the branch priority. Since the branch priority is obtained from the time-span tree and the note duration, it is redundant information, but it is differentiated in this paper for clearer explanation. qqqq 1234 d8 e8 e8 f8 0 1 4 1 1 3 4 2 Fig. 9. Matrix representation of time-span tree 3.1 Stepwise Rendering The 2nd and 3rd rows of the time-span-tree matrix in Fig. 10 are the same as in Fig. fig:matrix. When starting to render the time-span tree, one note "d32" is connected to the branch with the highest priority (Fig. 10(a)). In the next step, a note is rendered on the 4th branch, which has the 2nd highest priority. Branch 1 is to the left of branch 4, so the input to the Transformer model is "d32 l.". The pitch and duration of the rendered notes are determined by the Transformer model output. If the Transformer model output is "d16 f16" then the 1st row of the time-span-tree matrix will be "d16 - - f16" (Fig. 10(b)). The 2nd branch with the 3rd highest priority is then rendered (Fig. 10(c)). Finally, the 3rd branch with the lowest priority is rendered (Fig. 10(d)). The Transformer model may produce unexpected outputs from untrained inputs. In such a case, it may be difficult to proceed with the rendering process and our method multiplies the initial duration of one note by a randomly chosen value between 2 and 16 and restarts the rendering process. In our study, the Transformer model was implemented with the OpenNMTpy toolkit (ver. 2.0.0rc2) [11]. The default parameters were used because the estimation performance for rendering is constant regardless of the parameters. We confirmed that other translation networks, such as Seq2Seq model, can also learn stepwise rendering [15]. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 968 Melody-rendering Method Based on GTTM (b) h hq Rendering (d) wh Rendering (c) d32 l. → d16 f16. d16 l f16. → d8 e8 f16. qqh q Rendering qqqq (a) d8 e8 r f16. → d8 e8 e8 f8. d32 − − − 0 1 4 1 1 3 4 2 d16 − − f16 0 1 4 1 134 2 d8 e8 −f16 0 1 4 1 1 3 42 d8 e8 e8 f8 0 1 4 1 1 3 4 2 1 2 3 4 1 2 3 4 1 2 34 12 3 4 Fig. 10. Learning data for melody rendering (b) Time-span reduction � (c) ��� � (a) d32 −−− 0141 1342 d16 −−f16 0141 1342 d8 d8 f8 f8 0141 1342 1234 � Rendering system Transformer Finished rendering all branches? d32 l. d16 f16. ↓ ↓ Yes No �� �� � (d) (e) Fig. 11. Overview of proposed melodyrendering method 3.2 Procedure of Proposed Melody-rendering Method The input of our method is a time-span-tree matrix reduced to one note as well as rest positions and lengths. The procedure of our method is as follows (Fig. 11). (a) Generate a string for the Transformer model input from the time-span-tree matrix. (b) AstringisoutputfromtheTransformermodel. (c) Substitute the output string into the time-span-tree matrix. (d) If there is a branch to render, go back to (a) and repeat. (e) Return the rest to the original position and finish. In GTTM, the definition of rests in a time-span tree is ambiguous [12] However, the definition of maximum time-span states that rests are not included in the time-span [8]. Therefore, our method inserts rests at their original positions after rendering without rests. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 969