Looking Glass: Visualizing the Internal State of Music Models for Live Human-AI Improvisation

In live improvisation, musicians use visual cues to signal upcoming musical direction. However, many generative AI music models lack this visual communication. To address this challenge, Rachel worked with her advisors, along with PhD mentors and peers, to develop Looking Glass, an interactive visualization system designed to reveal the internal state of AI music models to enhance live human-AI improvisation.

The system projects graphics directly onto a keyboard, with each visual indicator aligned to its corresponding key to display the AI’s upcoming notes. A confidence score also shows how closely the performer’s real-time MIDI input aligns with the AI model’s learned musical distribution. Adjustable visualization settings allow musicians to choose how much attention they devote to the AI’s behavior versus their own improvisation.

Rachel evaluated the system in a user study with six experienced improvising musicians. The results suggest that the visualization of future notes supports clearer communication, enabling performers to anticipate the model’s direction without diverting attention from their instrument. While participants had mixed opinions about the confidence score, the study demonstrates the potential of exposing an AI model’s internal state to create more transparent and effective human-AI co-creation.