S2H2S: Modelling Expressive Piano Performances via Continuous Piano Key Motion Signals

Jingjing Tang, Shinichi Furuya, George Fazekas, Vincent Cheung

Primary Subject: Early Research

Some of the required materials for this paper do not exist: Video

Abstract:

We present a bidirectional framework, S2H2S, for modelling the relationship between piano performance audio and Hackkey signals—high-resolution, continuous measurements of vertical key position captured during performance. Unlike symbolic representations such as MIDI, Hackkey encodes fine-grained piano gesture dynamics, including key press depth, velocity, and release timing, offering a physically grounded perspective on expressive piano technique. The framework comprises two core components: sound-to-Hackkey (S2H) transcription and Hackkey-to-sound (H2S) synthesis. For both tasks, we adapt and fine-tune state-of-the-art neural models to predict continuous Hackkey trajectories from audio input and to synthesise expressive audio directly from Hackkey signals, respectively. The system demonstrates promising results on recordings of Hanon exercises, achieving accurate gesture reconstruction and faithful audio synthesis, supporting new use cases in expressive performance analysis, music performance synthesis, and music education.