Draw a doodle of a drum or saxophone to conjure up a multi-instrumental composition. Look into a webcam, talk, and watch your mouth bounce off the screen – the input to a series of delightfully goofy chain reactions.
This is what visitors to the MIT Lewis Music Library encounter when they interact with two new digital installations, “Doodle Tunes” and “Sounds from the Mouth,” created by 2022-23 Center for Art and Technology (CAST) Guest artist Andreas Refsgaard in collaboration with digital media and music technology librarian Caleb Hall. The residency was initiated by Avery Boddie, department head at Lewis Music Library, who recognized Refsgaard’s talent for revealing the playfulness of emerging technologies. The complexities of coding and machine learning can seem overwhelming to newcomers, but Refsgaard’s practice as a creative coder, interaction designer, and educator seeks to open the field to everyone. Spanning workshops, an artist talk, class visits, and an exhibition, the residency was infused with his unique sense of humor, a combination of lively eccentricity and laid-back relativity.
Machine Learning and the Arts with MIT CAST Guest Artist Andreas Refsgaard
learning through laughter
Copenhagen-based Refsgaard is a true machine learning maverick. “I’m interested in the ways we can express ourselves through code,” he explains. “I like to make unconventional connections between inputs and outputs, with the computer serving as a translator: a tool might let you play music with your eyes, or it might generate a love poem from a photo of a donkey.” Refsgaard’s particular spin on innovation isn’t about directly solving problems or launching world-changing startups. Instead, he simply seeks to “dig into what can be done,” providing accessible open source templates for new ideas and creative applications.
Programmed by Refsgaard and featuring a custom set of sounds created by Hall, “Doodle Tunes” and “Sounds from the Mouth” demonstrate how original compositions can be generated through a combination of spontaneous human gestures and algorithmically produced results. In “Doodle Tunes,” a machine learning algorithm is trained on a data set of drawings of different instruments: a piano, drums, bass, or saxophone. When the user draws one of these images on a touch screen, a sound is generated; the more instruments you add, the more complex the composition becomes. “Sounds from the Mouth” works through facial tracking and self-captured images. When the participant faces a webcam and opens their mouth, a self-contained snapshot is created that bounces off the notes of a piano. To try the projects yourself, scroll to the bottom of this article.
Libraries, unlimited
Saxophone screeches and digital drum hits aren’t the only sounds emitted from the areas where the projects are installed. “My office is close by,” says Hall. “So when I suddenly hear laughter, I know exactly what’s going on.” This new sonic dimension of Lewis Music Library fits with the spirit of the setting as a whole: designed as a campus hub for audio experimentation, the library was never intended to be completely silent. The Refsgaard residency exemplifies a new emphasis on progressive programming spearheaded by Boddie, as the library’s strategy shifts toward a focus on digital collections and music technology.
“In addition to serving as a space for quiet study and access to physical resources, we want the library to be a place where patrons meet, collaborate, and explore together,” says Boddie. “This residency was very successful in that sense. Through the workshops, we were able to connect people from across the MIT community and their unique disciplines. We had people from Sloan School of Management, from Schwarzman College of Computing, from Music and Theater Arts, all working together, getting dirty, creating tools that sometimes worked… and sometimes didn’t.”
mistake and serendipity
The integration of error is a key quality of Refgaard’s work. The occasional glitch is part of the art and also serves to gently undermine the hype around the AI; An algorithm is only as good as its data set, and that set is influenced by human bias and oversight. During a public artist talk, “Machine Learning and the Arts,” audience members were introduced to Refsgaard’s offbeat artistic paradigm, featured with projects like booksby.ai (an online bookstore of science fiction novels produced by AI), IS IT FUNKY? (an attempt to distinguish between “fun” and “boring” images), and eye driver (an interface to play music through eye movements and facial gestures). Flaws in the exhibit’s installations were candidly admitted (admittedly, “Doodle Tunes” occasionally mistakes a drawing of a saxophone for a squirrel), and Refsgaard encouraged audience members to suggest possible improvements.
This open-minded attitude set the tone for the “Art, Algorithms, and Artificial Intelligence” and “Machine Learning for Interaction Designers” workshops, aimed at beginners and curious experts alike. Refsgaard’s visits to music technology classes explored the ways in which human creativity could be amplified by machine learning and how to navigate the sliding scale between artistic intent and unexpected results. “As I see it, success is when the participants interact with the material and come up with new ideas. The first step in learning is to understand what is being taught; the next is to apply that understanding in ways the teacher could not have foreseen.”
uncertainty and opportunity
Refsgaard’s work exemplifies some of the core values and fundamental questions for the evolution of MIT Libraries: issues of digitization, computing, and open access. By choosing to make his joyous demos freely available, he relinquishes ownership of his ideas; a machine learning model could serve as a learning device for a student, and a corporation could monetize it just as well. For Refsgaard, play is a way to engage with the ethical implications of emerging technologies, and Hall found himself grappling with these questions in the process of creating the sounds for the two installations. “If I wrote the sound samples, but someone else arranged them as a composition, who owns the music? Or does AI own the music? It’s an incredibly exciting time to be working in music technology; We are entering uncharted territory.”
For Refsgaard, uncertainty is the secret ingredient in his algorithmic art. “I like to do things where the end result surprises me,” she says. “I look for that sweet spot between something familiar and something unexpected.” As he explains, too much surprise just equals noise, but there’s something joyful about the possibility that a machine might mistake a saxophone for a squirrel. The task of a creative coder is to continually adjust the relationship between human and machine capabilities to find and follow the music.
“scribble melodies” and “sounds of the mouth” are on view at the MIT Lewis Music Library (14E-109) through December 20. Click on the links to interact with the projects online.
