P2-16: MusiQAl: Music Question Answering through Audio-Video fusion
Anna-Maria Christodoulou, Kyrre Glette, Olivier Sergei Lartillot, Alexander Refsum Jensenius
Subjects: TISMIR ; Knowledge-driven approaches to MIR ; Novel datasets and use cases ; Applications ; Music videos, multimodal music systems ; Machine learning/artificial intelligence for music ; MIR fundamentals and methodology ; Digital libraries and archives ; Evaluation, datasets, and reproducibility ; Multimodality
Presented In-person
4-minute short-format presentation
Music Question--Answering (MQA) is a machine learning task where a computational system analyzes and answers questions about music-related data. Traditional methods prioritize audio, overlooking visual and embodied aspects crucial to music performance understanding. We introduce MusiQAl, a multimodal dataset of 310 music performance videos and 11,793 human-annotated question--answer pairs, spanning diverse musical traditions and styles. Grounded in musicology and music psychology, MusiQAl emphasizes multimodal reasoning, causal inference, and cross-cultural understanding of performer-music interactions. We benchmark AVST and LAVISH on MusiQAl, revealing strengths and limitations, underscoring the importance of integrating multimodal learning and domain expertise to advance MQA and Music Information Retrieval (MIR).