Real-time Visualizer for Beats and Scratches of Breaking DJ Performances

Real-time Visualizer for Beats and Scratches of Breaking DJ Performances Masatoshi Hamanaka1 RIKEN, Tokyo, Japan [email protected] Abstract. Breaking is a dance duel between two dancers to music improvised by a DJ. Since breaking was chosen as an official competition at the Paris Olympics, many people have begun to watch the competitions on television. When a dancer freezes in time with a beat or scratch in breaking, it is called “kill the beat," and when it is successful, the audience gets very excited. However, TV viewers may have difficulty hearing the music due to the announcer’s or commentator’s voice, making it difficult to judge if the kill the beat is successful or not. In addition, some in the audience may be hard of hearing. Therefore, we developed a visualizer for the beats and scratches of breaking DJ performances. Keywords: Breaking DJ performance ·Visualizer ·Beat ·Scratch. 1Introduction Breaking features two dancers breakdancing against each other to the rhythm of music played by a DJ (disc jockey). The DJ uses two turntables to play multiple musical pieces while switching between them. The DJ then adds accents to the music by scratching the turntables. Freezing while performing tricks, such as inverted stands, in perfect time with the rhythm generated by the DJ is called “kill the beat" and is the highlight of competitions, as it requires a high level of skill. Most of the audience at a breaking venue have experienced breaking, and they get very excited when they see a successful kill the beat. However, viewers watching breaking videos on television have difficulty hearing the music due to the announcer’s and commentator’s voices, making it difficult to judge whether a kill the beat is a success, and some viewers are also hard of hearing. Therefore, we developed a visualizer for breaking DJ performances that displays the beat of the sound source being played and scratches played by the DJ. Several methods have been proposed for automatically generating a set list for DJs to prepare before remixing [1,6,10]. An automated DJ-mixing system was also proposed [7] as well as a method for smoothly crossfading from one All rights remain with the authors under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Proc. of the 17th Int. Symposium on Computer Music Multidisciplinary Research, London, United Kingdom, 2025 Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 1031 Masatoshi Hamanaka piece to another [11]. While these studies [1,6,10,7,11] intended to automate the DJ’s work or assist the DJ, our study is intended to assist the breaking audience through visualization. Beat visualization helps the audience understand if a beat matches the timing of the moves. Visualization can also enable the hearing impaired to understand the timing of the beat and enjoy breaking. Our visualizer was used in 2024 at the All Japan Breaking Championships recorded by the Japan Broadcasting Corporation (NHK in Japanese) (Fig. 1). Fig. 1. Beat and Scratch Visualizer 2BeatandScratchVisualizer There are five problems in visualizing the beats and scratches created by a DJ in real time. Each of these problems and their solutions are described as follows. No network: To execute the visualizer, it was necessary to connect the DJ table at the rear of the stage at NHK Hall, the competition venue, to the NHK broadcast room over a distance of more than 100 meters via a network. However, NHK Hall did not have network facilities. Therefore, we built a 10G network by connecting a small-form-factor pluggable transceiver to a standard LEMO [8] fiber optic cable for the TV camera. Difficulty of predicting beats: Predicting beats in real time is difficult because DJs change musical pieces one after another. Therefore, we receive all sound sources used by DJs, analyze them, and generate beat-visualization videos. The sound source and video can then be played synchronously by creating a link between the sound source and video. When a DJ changes the music-playback speed (turntable rotation speed), the video-playback speed also changes in sync. Appropriate musically: The rhythm-visualization video to be created consists of long bars for strong beats and short bars for weak beats and scrolls vertically with time (Fig. 1). The strong and weak beats can be obtained from AI analysis. However, simple AI analysis sometimes results in the strong and weak beats being reversed, depending on the timbre of the instrument or other factors. Therefore, we added a musical analysis to solve the reversal problem [9,2–5]. Video and signal delays: The broadcast video produced by NHK included delays as it passed through selectors that integrate multiple cameras and switchers that superimpose information, etc (Fig. 2(a)). The distance between the cameras and the broadcast room at NHK Hall, the venue of the championships, was more Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 1032 Real-time Visualizer for Beats and Scratches than 100 meters, which caused transmission delays. However, our real-time beat and scratch visualizer also needed to send information from the DJ booth to the broadcast room, which caused delays. Due to these delays, the timing between the dance video and beat-visualization video is shifted. Therefore, we can adjust the position of the beat-visualization video by adding a line indicating the current time on the video later in the program so that the timing of the dance video and beat-visualization video are aligned. Animation at current timing: Adding an effect, such as a color change or animation, when bars indicating whether the beat is strong or weak pass the current timing makes the timing easier for the viewer to understand. However, a video with effects added in advance cannot be created because the timing and position change depending on the delay in the previous item. Therefore, we developed a program that analyzes the beat-visualization video being played, reads the bar positions, and adds effects at the appropriate timing. Switcher Audio mixer Inserter HDMI out DJ’s computer Rhythm-visualization video Keyer 100 m Dance footage broadcast by NHK Video cameras Microphones Turntable B Turntable A Mixer Audio In USB Audio/MIDI Interface MIDI OUT SDI capture SDI converter USB (MIDI) HDMI MIDI Interface SDI USB DJ’s computer Mac mini for scratch detection Mac Studio for effect generation Windows for software switcher SDI USB HDMI Fill video Key video USB (a) (b) Fig. 2. (a)Equipment-connection and transmission delays (b)Overview of visualizer 3 Implementation Scratches are detected by the Mac Mini using turntable rotation information and DJ mixer fader information (Fig. 2(b)). Detection of turntable rotation: Two turntables are connected to the DJ mixer, and DJ software can be used to set the pieces assigned to each turntable. When using DJ software, a vinyl record in which sine waves with different phases are recorded on the left and right sides is placed on the turntable. The DJ software then receives the rotation information of the turntable from the digital vinyl system (DVS) signal generated by the vinyl record. Because the DVS signal is an analog signal, we branch it out, receive it at the audio interface, and analyze it to determine the revolutions per minute of both the left and right turntables. Specifically, accelerating the turntable increases the frequency, and decelerating the turntable decreases it. Because the DVS signal is a stereo signal and is out of phase between left and right, it can also detect whether the rotation is forward or reverse depending on which waveform arrives first. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 1033 Masatoshi Hamanaka Scratch detection: DJ mixers have a crossfader, which is used for swapping between two pieces or for adding scratching sounds. Scratching is a technique in which a record is played on one turntable while a record on the other turntable is scratched during the playback of a certain piece to emphasize the beat with the sound generated. The crossfader can then be moved slightly to the turntable on the side that is being scratched to mix both the playback and scratch sounds. Therefore, to detect scratches, we need to detect not only the rotation of the turntable but also the position of the crossfader. Many models of DJ mixers that can be connected to a computer via a USB cable use MIDI signals for communication. Some DJ mixer models that are equipped with two USB ports for DJ alternation can receive the USB signal flowing through the port in use on the other USB port. Therefore, the MIDI signal is analyzed, and the crossfader position is extracted. 4ExperimentalResults Equipment and transmission delays are dependent on the hall and broadcasting facilities, so adjustments were made on site after the equipment was installed in NHK Hall. The breaking TV video received from NHK was a time-codesynchronized switching of multiple camera images. However, the sound signals from the stage, the emcee’s voice on stage, and the DJ’s performance of a song were picked up by microphones. Video and sound were lip-synchronized by NHK using the voice and video of the emcee during rehearsal. We recorded the video after superimposing beat and scratch visualization on that video. While slowing down and playing back the recorded video, we checked the beat timing of the acoustic signal and timing of the bar passing the white line representing the current time and found that the visualized video was delayed by two frames at 60 frames per second. The white line representing the present time was moved 20 pixels downward and recorded again. As a result, the passage of the bars in the visualization video coincided with the timing of the acoustic signal in the NHK dance video. When twenty people who viewed the recorded video were asked their opinions, 18 said they enjoyed watching it, and 14 said kill the beat was easy to understand. 5Conclusion The beat and scratch visualizer we constructed was used at the All Japan Breaking Championships in February 2024. A video was recorded using a sound source and a video visualizing the sound source after superimposing beat and scratch visualization on the video. Slow playback of the recorded video showed a discrepancy between the timing of the sound in the broadcast video and timing of the beat visualization. Therefore, the current point of the visualization video was adjusted to eliminate the misalignment. We plan to conduct more detailed experiments. We plan to conduct more detailed experiments. Disclosure of Interests The author have no competing interests. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 1034 Real-time Visualizer for Beats and Scratches References 1. Bittner, R.M., Gu, M., Hernandez, G., Humphrey, E.J., Jehan, T., McCurry, H., Montecchio, N.: Automatic playlist sequencing and transitions. In: Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR2017). pp. 442–448 (2017) 2. Hamanaka, M., Hirata, K., Tojo, S.: Automatic generation of metrical structure based on gttm. In: Proceedings of the 2005 International Computer Music Conference (ICMC2005). pp. 53–56 (2005) 3. Hamanaka, M., Hirata, K., Tojo, S.: Implementing a generative theory of tonal music. Journal of New Music Research 35(4), 249–277 (2006). https://doi.org/10.1080/09298210701563238, https://doi.org/10.1080/09298210701563238 4. Hamanaka, M., Hirata, K., Tojo, S.: DeepGTTM-II: Automatic Generation of Metrical Structure Based on Deep Learning Technique. In: Proceedings of 13th Sound and Music Computing Conference (SMC2016) (Aug 2017). https://doi.org/10.5281/zenodo.851229, https://doi.org/10.5281/zenodo.851229 5. Hamanaka, M., Hirata, K., Tojo, S.: deepgttm-iv: Deep learning based time-span tree analyzer of gttm. In: Proceedings of the 2023 of the 16th International Symposium on on Computer Music Multidisciplinary Research (CMMR2023). pp. 611– 618 (2023). https://doi.org/10.5281/zenodo.10114163 6. Hirai, T., Doi, H., Morishima, S.: Musicmixer: Automatic dj system considering beat and latent topic similarity. In: Proceedings of 22nd International Conference of MultiMedia Modeling (MMM2016). pp. 698–709. Springer International Publishing, Cham (2016) 7. Hiromi Ishizaki, Keiichiro Hoashi, Y.T.: Full-automatic dj mixing system with optimal tempo adjustment based on measurement function of user discomfort. In: Proceedings of the 10th International Society for Music Information Retrieval Conference (ISMIR2009). pp. 135–140 (2009) 8. LEMO: About us | LEMO Connectors and cables — lemo.com. https://www.lemo.com/our-world/company/about-us, [Accessed 07-03-2024] 9. Lerdahl, F., Jackendoff, R.: A generative theory of tonal music. The MIT Press, Cambridge. MA (1983) 10. Parera, J.: Dj codo nudo: a novel method for seamless transition between songs for electronic music. Master’s thesis, Universitat PompeuFabra, Barcelona (2016) 11. Robinson, K., Brown, D.: Automated time-frequency domain audio crossfades using graph cuts. In: Proceedings of the 20th International Society for Music Information Retrieval Conference (ISMIR2019) Late-Breaking/Demo (2019) Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 1035