We propose MusicViz, a graphical animation system that visualizes decision processes in transformer-based music models. The purpose of our system is to make human-artificial intelligence (AI) collaboration more transparent and intuitive. By combining piano-roll and staff notation, the system displays notes as rectangles linked by weighted arcs representing attention between note tokens. This visualization helps musicians grasp the model’s “thought process” by revealing detected patterns such as motifs and longer sequences. Interactive features allow users to zoom, scroll through time, and toggle attention weights, enabling detailed exploration of the model’s behavior. By revealing the AI's internal focus, musicians can adapt their input to guide generated responses, fostering more effective collaboration. MusicViz addresses the challenge of interpreting complex transformer computations in a human-friendly way. The current offline prototype lays the groundwork for future real-time generative music visualization, with potential applications in live performance and improvisation.