If the red dot at the top left of the video does not become green when your hand moves into view, I'm afraid the app doesn't work in your browser yet. You could try your computer: the app works fine in Chrome, Firefox and Safari on my MacBook.
For hundreds of years, musicians were trained to sight-read music using a solmization method that revolved around the Guidonian hand. Simplifying things a bit, the idea is that every joint of the hand corresponds to a note. Singers would touch the joint of the note they were singing with their thumb. In this way, they learned to associate notes with a physical gesture — a deeply embodied way of learning solmization. This is all quite similar to how instrumentalist associate notes with a position on their instrument. In fact, hands could even be used as instruments themselves: we know that choir masters could use the same gestures to indicate which notes the singers were supposed to sing.
Manual Music is an experiment by Bas Cornelissen that tries to turn the hand into an instrument. The project revolves around a model developed by Google that can detect the position of a hand in images and videos. The difficult part is to recognize the different gestures. The project is in an early stage, but already allows you to 'play' the Guidonian hand. The detection method used is however not very reliable yet (see updates below). Interested in joining the project? Please get in touch!
You can find all code on GitHub. Built using Mediapipe Solutions, Fresh, Deno, Tailwind, Preact and more.
Copyright © Bas Cornelissen, 2024.