Learning Machine Knit Minput Intertextiles

Learning Machine Knit Minput Intertextiles VICTOR SHEPARDSON, Intelligent Instruments Lab, University of Iceland SOPHIE SKACH, Intelligent Instruments Lab, University of Iceland Additional Key Words and Phrases: e-textiles, no-input mixing, knit structures, interactive machine learning ACM Reference Format: 1 PROGRAM NOTES What lies among these loops of copper, wool and steel? Skin, yarn and wire form a circuit, machine precision meets somatic sense, and tactility elaborates algorithmic entanglements. In this performance, e-textile pieces are patched into a no-input mixer, while player uses interactive machine learning tools to embroider sonic layers onto a repeated gesture. Analog audio signals from a no-input mixing board are routed through a piezoresistive mat which has been knitted from wool and steel yarns. The performer first develops a patch on the no-input mixing board to sound a two-handed pressing gesture on the mat. Then, the they alternate between performing the gesture and refining mappings between a machine listing algorithm and additional layers of sound using the anguilla interactive machine learning package. 2 PROJECT DESCRIPTION Electronic Textiles, or e-textiles, have shown to be a suitable, flexible, playful, accessible, and otherwise versatile design material for soft interfaces for musical expression. Often, textile sensors are aimed to provide accurate measurements to replace more rigid midi devices. The very material properties of textile surfaces, their structure, yarn quality, interactive behaviour is not accounted for in such examples. Here, instead of redesigning an existing instrument with sensorised textiles, we propose a textile surface itself as an experimental electronic instrument, by incorporating it into the circuit of a no-input mixing desk. 2.1 Background This music performance marries three disciplines: electronic textiles (e-textiles), no-input mixing, and interactive machine learning. 2.1.1 E-textiles. Textiles are an interface we have been familiar with for thousands of years, often in the form of, but not bound to clothing. Seen as an extension of our skin, they are used to culturally and socially express ourselves [ 6 ]. Embedding conductive materials like metal fibres, or carbon coated yarns has enabled textiles to become sensing surfaces. Research in this area has been explored for several decades, and has become a fundamental part of wearable Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s). Victor Shepardson and Sophie Skach. 2024. Learning Machine Knit Minput Intertextiles. 1, 1 (September 2024), 5 pages. Music Proceedings of the International Conference on New Interfaces for Musical Expression NIME’24, 4–6 September, 2024, Utrecht, The Netherlands 2 Victor Shepardson and Sophie Skach Fig. 1. Playing the knit mat with pressure from a hand computing [ 5 ]. While often being proposed for applications in health care or sports, textile sensing networks have also found its place in performance art [ 7 ], including music and sound art [ 8 ]. Also within the NIME community, e-textiles in various shapes and textures have been introduced as soft, wearable instruments. For example, a high-resolution textile pressure sensor matrix by Donneaud et al [ 1 ], or a knitted, scarf-like keyboard by Wicaksono et al [ 10 , 11 ] have demonstrated that textiles are reliable sensing surfaces and facilitate new types of sonic interaction. 2.1.2 No-input Mixing. This is the practice of looping the outputs of a mixing desk back to the inputs, creating a positive feedback loop and causing the mixer to self oscillate. It is a widespread practice in experimental music, with antecedents going back to the 1960s. According to Mudd, the term “no-input mixing” was coined by Toshimaru Nakamura, one of its most prominent exponents [4]. When a mixer is patched into itself, the circuit formed will tend to have some resonant frequency, and the result will be a tone at that frequency. But this basic story doesn’t capture the essence of no-input mixing. With careful navigation of the mixer controls, a player can surface exactly the nonlinearities and imperfections which remain hidden when the mixer is used for its original purpose, leading into a wilderness of sputtering, clicking, hissing and squawking. These states are often incredibly delicate and sensitive to the slightest motion of a knob. We wanted to explore the materiality of e-textiles beyond being soft and stretchy sensors, and we found that incorporating e-textile pieces into a no-input mixer could create an extremely detailed relationship between sound and touch which brings out the particularities of each object. Learning Machine Knit Minput Intertextiles 3 Fig. 2. Made from scratch: flexible connectors and soft cables for the No-Input mixer 2.1.3 Interactive Machine Learning (IML). This is a blanket term for systems which allow live construction of mappings (e.g. from gesture to sound) from small, manually created datasets. This can allow creating mappings by example which would be far more difficult to program explicitly [ 9 ]. One of the most widely influential tools is Fiebrink’s Wekinator [3], which has been used extensively for musical instrument design [2]. We are interested in IML for discerning gesture from the raw sound of the no-input textiles, and then mapping it back onto additional layers of sound. In this performance, we use the anguilla 1 IML package to build mappings from gesture to sound. Anguilla is a package for interactive supervised learning built around nearest neighbor search algorithms. It eschews separate data collection and training steps while exposing a hackable Python implementation to support rapid prototyping. 2.2 Instrument Design The “Knit Minput Intertextiles" consist of a No-input mixer and a soft e-textile mat interface connected to it through textile ‘cables’. The mat is a machine knitted fabric made from wool and stainless steel yarn. 2.2.1 Knitting Pattern. The pattern shown on the mat, seen in Fig.3, is modified from a historical ornamental design of the 18th Century, originating from the region this interface, as is the woollen yarn it is knitted with. It was produced on a computerised flatbed knitting machine (a “Kniterate" model) in a technique called positive-negative jacquard. As the name suggests, this knitting technique creates the colour reversed image of the knitted fabric on the backside with the yarn alternating between needle beds. Therefore, the mat can be used from both sides. 2.2.2 Conductive Yarn. To be able to sonically enhance the knitted structure, turning the textile into an e-textile, conductive yarn was used alongside wool yarn. A stainless steel yarn in light grey colour is used, spun together with a locally sourced, non-conductive sheep’s wool, forming the substrate yarn and creating the contrasting colour effect. 2.2.3 Textile Cables. To connect the mat to the No-Input mixer, soft cables were costum-made in various lengths and with different ends, as seen in Figure 2. This is to accommodate material appropriate connectors, especially to not damage the textile interface. Additionally, crocheted and knitted pleats, seen in the middle and right images in Fig.2, are 1https://github.com/Intelligent-Instruments-Lab/anguilla 4 Victor Shepardson and Sophie Skach used as ‘cables’ that vary in resistance depending on how they are folded or unfolded, enhancing the effects of the sonic textile deformation. A combination of stainless steel and silver coated yarn is integrated in these. 2.3 Proposed Performance This performance is designed to bring out affinities between the e-textiles, the no-input mixer, and the machine learning algorithms. It is semi-improvised and different each time, but always structured in two parts. In Part I, the performer builds a patch on the no input mixer, live in front of the audience. The patch is tweaked to respond to a particular gesture of spreading both hands on the mat and slowly applying body weight. For this part, the direct sound of the mixer is used. The patch should be tuned so that there is silence while the textile mat is not being touched. In Part II, the performer alternates between repeating the gesture at an improvised but generally slowing pace, and exploring a newly added layer of sound. In each iteration, the machine listening system collects a sequence of input points as the main gesture is performed. Then, the performer activates a new sound layer with a random mapping from these input points to synthesizer parameters. Next, the performer explores this random mapping, collecting output points where the sound is compelling to them. Finally, these sets of input and output points form the final mapping through the IML system, and the gesture is performed once more with the newly constructed sound layer, starting the cycle again. As the gesture is performed, all layers of sound can be heard mixed together. As the sound becomes more complex, the performer finds more to listen to, and the gesture slows down and develops finer nuance in the distribution of weight and position of hands. 3 PERFORMANCE NOTES The performance lasts about 8-12 minutes. Table space with an electrically insulating surface of at least 60x120cm is needed to set up the mixer, textile mats, laptop and soundcard. Performers will provide the soundcard, which has 1/4" balanced phono outputs. Quadrophonic output is ideal but stereo is acceptable. The audience should have a good view of the table surface. This can be achieved either by video projection, or in a small venue with elevated audience. If video projection is used, a camera will be provided by the performers to capture the surface of the mat, composited in the laptop, and sent via HDMI. Two power outlets are needed for laptop and mixing board. The intended sound level is high, similar to noise music. However a lower level is acceptable. The piece could work in either a concert hall or club venue, but would ideally be programmed next to work with a similarly loud level and/or noise aesthetic. 4 MEDIA LINKS A playlist of videos demonstrating the no-input textiles can be found here: https://www.youtube.com/watch?v= VT8Ht0lf_F4&list=PLK7FLipUDWEE1DbcmiXQZJ_Az9D3iGMDN&pp=iAQB. See proceedings for video documentation of the proposed performance. ACKNOWLEDGMENTS The authors would like to thank The Icelandic Textile Center. The Intelligent Instruments Lab is supported by the European Research Council (ERC) as part of the Intelligent Instruments project (INTENT), under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 101001848). Learning Machine Knit Minput Intertextiles 5 Fig. 3. Performance Setup ETHICAL STANDARDS This project was funded by an ERC grant. No humans or animals were research subjects in this work. REFERENCES [1] Maurin Donneaud, Cedric Honnet, and Paul Strohmeier. 2017. Designing a multi-touch etextile for music performances.. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME), Vol. 17. 7–12. [2] Rebecca Fiebrink and Laetitia Sonami. 2020. Reflections on Eight Years of Instrument Creation with Machine Learning. In Proceedings of the International Conference on New Interfaces for Musical Expression, Romain Michon and Franziska Schroeder (Eds.). Birmingham City University, Birmingham, UK, 237–242. https://www.nime.org/proceedings/2020/nime2020_paper45.pdf ISSN: 2220-4806. [3] Rebecca Anne Fiebrink. 2011. Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. Ph.D. Dissertation. Princeton University. [4] Tom Mudd and Akira Brown. 2023. Contrasting approaches to the no-input mixer. Proceedings of the International Conference on New Interfaces for Musical Expression (May 2023), 402–408. http://nime.org/proceedings/2023/nime2023_56.pdf Place: Mexico City, Mexico. [5] E.R. Post and M. Orth. 1997. Smart fabric, or "wearable clothing". In Digest of Papers. First International Symposium on Wearable Computers. 167–168. https://doi.org/10.1109/ISWC.1997.629937 [6] Susan Elizabeth Ryan. 2014. Garments of Paradise: Wearable Discourse in the Digital Age. The MIT Press. https://doi.org/10.7551/mitpress/8873.001. 0001 [7] Sophie Skach, Anna Xambó, Luca Turchet, Ariane Stolfi, Rebecca Stewart, and Mathieu Barthet. 2018. Embodied Interactions with E-Textiles and the Internet of Sounds for Performing Arts. In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction (TEI ’18). Association for Computing Machinery, New York, NY, USA, 80–87. https://doi.org/10.1145/3173225.3173272 [8] Rebecca Stewart. 2019. Cords and chords: exploring the role of E-textiles in computational audio. Frontiers in ICT 6 (2019), 1–12. https: //doi.org/10.3389/fict.2019.00002 [9] Federico Ghelli Visi and Atau Tanaka. 2021. Interactive machine learning of musical gesture. Handbook of artificial intelligence for music: Foundations, advanced approaches, and developments for creativity (2021), 771–798. [10] Irmandy Wicaksono and Joseph Paradiso. 2020. KnittedKeyboard: Digital knitting of electronic textile musical controllers. In Proceedings of the International Conference on New Interfaces for Musical Expression. Birmingham City University, Birmingham, UK, 323–326. [11] Irmandy Wicaksono and Joseph A Paradiso. 2017. Fabrickeyboard: multimodal textile sensate media as an expressive and deformable musical interface.. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME), Vol. 17. 348–353.