Listening to music is basically the interpretation of sensations, whereas making music requires the performer to actively create content. Hence, music-making requires a closed feedback loop to compare and evaluate the created sound with the intended sound. This becomes a challenging task for those with hearing disabilities who are nevertheless interested in learning to play an instrument.

We developed Music Sensory Substitution (MuSS) Bits, small wireless sensor-display pairs, to provide real-time feedback for sound coming from various audio sources, such as an instrument, a digital device or from the environment. Using sensory substitution, MuSS-Bits translate the audio signal into visual and haptic feedback.


  • Benjamin Petry, Thavishi Illandara, Suranga Nanayakkara. MuSS-Bits: Sensor-Display Blocks for Deaf People to Explore Musical Sounds. In Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction (OzCHI '16). ACM, New York, NY, USA, 72-80. [PDF] [DOI]
  • Benjamin Petry, Thavishi Illandara, Juan Pablo Forero, Suranga Nanayakkara. Ad-Hoc Access to Musical Sound for Deaf Individuals. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '16). ACM, New York, NY, USA, 285-286. [PDF] [DOI ], [Poster]
  • Benjamin Petry, Jochen Huber and Suranga Nanayakkara. Scaffolding the Music Listening and Music Making Experience for the Deaf. In Assistive Augmentation. Springer. Cognitive Science and Technology Series. (to appear).

Press and Awards