Senior Thesis: Using AI to Write the Score of Wesleyan

For his senior thesis, Ellington Davis ’25 set out to create a musical score of Wesleyan, comprised of the sounds that make campus feel familiar. He defined and isolated the chirps of birds, whirs of heating units, murmurs of conversations between classmates, and remote sirens from passing emergency vehicles, among others. Then he took his project a step further using contemporary tools, like artificial intelligence. 

Davis, a music and Science and Technology Studies double-major, trained a machine-learning algorithm, through open-source software, to match motion to sounds and images collected from campus for his senior thesis project, “Wherepeople: The Wild, Wild Urban.” For example, if a subject moved an arm in front of the motion-capture camera Davis used, the AI model located a sound and image that most closely aligned with the motion. Then the system projected the corresponding image on screen and played the corresponding sounds aloud. 

To train the model, he first spent hours walking through campus and listening for latent sounds. Once he felt he had listened enough, he began to record and photograph what he observed. He documented a wide range of inputs both on campus and off, from construction on the New Science Building to steam systems in the Freeman Athletic Center to a biofuel plant in Southington, Conn. that processes Wesleyan’s food waste. 

“I think what was really exciting about going around and documenting all this is that it's a way of seeing your space,” Davis said. “If you can listen and you can hear things and wonder, why are those things happening and where are they coming from? It’s also, how does that thing actually come to be? Sound is a way of questioning and exploring.” 

Next, he assigned each input an integer to serve as an anchoring point for the software in his score. “[The program] can turn motion into data and make sense of it very easily,” Davis said. “It’s like a guitar’s pedal board, except all of the pedals are math.” 

Once completed, Davis demonstrated his creation at a recital with his band STATIC on April 12 at Hewitt Workshop. 

“This system was able to create a live, constantly changing, non-sovereign soundscape, based on the score of motion in the space it was viewing, made possible by training,” Davis wrote in his thesis paper. “The webcam (set up from my computer) read the audience which attended my thesis performance and Hewitt Workshop as a score—transforming each attendee into a soundwalker.” 

His project was inspired by readymade art practices in experimental music, like the late John Cage, an influential experimental composer who was involved with the University’s Music Department for over 50 years, and his score “Atlas Eclipticalis,” which superimposed musical staves over star charts. Davis is also interested in the "tendency of urban dwelling humans to tune out” everyday sounds and creations that can be seen as mundane, he said. He wanted to make sense of the inputs we may take for granted. To do this, he used anthropological and musical sign systems, in the form of his listening walks and semiotics. 

“I wanted to create a system that could compress all of this infrastructure that's a part of this disenchanted landscape and make a sort of enchantment out of it, where you can't ignore all of these sounds happening at once,” Davis said. 

After graduation, Davis plans to return to the Bay Area to work with political organizations and work on urban farms.