Building the MIDI Gesture Controller: Process Archive (April 2025 - May 2025)
Project Proposal April 9, 2025
Hand-Gesture Controlled Musical Interface
Theme: A wearable and gesture-based musical controller that allows improvisers and songwriters to play and explore chord progressions using only their fingers and hand movement. The prototype functions as both an instrument and a composition tool, designed for on-the-go, real-time improvisation, or live performance. The goal is to create a system that allows expressive and intuitive musical interaction without traditional instruments.
Typology: Musical composition tool, wearable performance interface
Interaction:
- Flex sensors mounted between two velcro finger bands (elastic + velcro straps at the base and tip of each finger)
- Flexing of fingers triggers specific chords mapped to a selected key
- Sensor board is mounted on the wrist with an elastic velcro strap
- The onboard accelerometer + gyroscope is used to detect downward hand motion, tilt, or rotation to modify chord voicings, inversions, or dynamics
Materials:
- 5x Flex Sensors (2.2")
- 5x 10k Resistors
- Jumper wires (Adafruit silicone-coated)
- Elastic + velcro strips (for finger and wrist mounting)
- Microcontroller/sensor board with 5 analog inputs
Software/Frameworks:
- Python with mido or rtmidi to send MIDI messages to a DAW
- DAW integration (Reaper)
Research Aspect:
- Understanding and implementing analog voltage circuits for multiple flex sensors
- Real-time mapping of finger gestures to MIDI chord structures
- Combining analog sensor data with readings to control musical parameters
Prototyping Plan (April 16, 2025)
This project explores a wearable music composition tool that maps finger gestures to functional harmony. The system uses finger-mounted IMUs, a central sensor board, and Python-based MIDI control to trigger chords in a DAW. The interface is designed as a compositional aid for songwriters and improvisers, especially those interested in capturing spontaneous ideas in a physical and experimental way.
What I’ve Already Done
Music Generation Research and First Scripts
I began the project by researching Python libraries for music generation and theory handling. I settled on Mingus for its theory-based chord and scale logic, and Mido for MIDI output. I started by generating I chords in various keys using Mingus, then extended this to all diatonic triads and sevenths(I–VII). I added key selection to the script and verified that Mingus and Mido differ in how they interpret octave numbers (Mingus’s octave 4 corresponds to Mido’s octave 3).
I consulted with a composer to determine what kind of output and control would be most helpful in a real creative workflow. Based on that input, I designed chord generation logic that accounts for proper voice leading and octave alignment.
MIDI Output and DAW Integration
After establishing basic theory logic, I shifted to output. I read through Mido’s documentation to understand real-time MIDI messaging, including how to open and name virtual output ports. I selected Studio One as the DAW for this project, based on recommendations from composers and its compatibility with virtual MIDI input.
I created a virtual MIDI port using Mido’s virtual=True parameter and configured Studio One to receive MIDI input from this source. Once routing was working, I sent a basic note_on message from Python and confirmed chord playback in the DAW.
From there, I experimented with sending chords by looping through lists of notes and sending multiple note_on messages. I built a terminal interface that allows a user to select the next chord interactively. I then attempted to implement logic to turn off the previous chord (note_off) before triggering a new one. Still working on that.
Hardware Preparation
I ordered and received two Adafruit LSM6DS3TR-C IMUs, one PCA9548 I2C multiplexer, and the necessary STEMMA QT/Qwiic cables. I also purchased elastic bands and velcro for attaching the sensors to my hand and fingers. After mounting the hardware to the elastic and velcro bands, I fitted the system on my wrist and index/middle fingers to begin ergonomic and positional testing.
What I Need to Do Next
Phase 1: IMU Data Testing (Sensor Board)
I will begin by using the built-in IMU on the sensor board to stream raw orientation data over serial. The goal is to view accelerometer and gyroscope readings in the terminal and identify how “hand up” vs. “hand down” gestures are represented numerically. I am not yet building gesture logic at this stage, just gathering and analyzing data.
Phase 2: IMU Data Testing (Index Finger)
Next, I’ll connect the IMU mounted to the index finger and repeat the same logging process. For this sensor, I’m focused on vertical motion only (finger up vs. finger down). The purpose is to determine thresholds or angle ranges that reliably distinguish those two states.
Phase 3: Map Gestures to Chords (Two Fingers)
With the index finger mapped, I’ll implement logic to trigger the I chord when the index finger is down. Then I’ll add the middle finger, and test a system where index + middle down triggers the II chord. This will serve as a functional test for both the gesture recognition and the mapping logic.
Phase 4: Scale Hardware and Chord Vocabulary
After confirming the approach works with two fingers, I’ll order and integrate three additional mini IMUs to cover all five fingers. If IMUs are not successful, I will order the flex sensors at this point. This will allow me to map all seven diatonic chords (I–VII) in a given key to different finger combinations. The logic for this will be modular and expandable, based on the previous finger-based tests.
Phase 5: Composer Feedback
Once basic functionality is in place, I’ll invite one or two composers to try the system and provide feedback. This stage will focus on:
The intuitiveness of the gesture mappings
How it feels to physically play chords
Whether the interface feels like a creative or inspiring tool
Phase 6: Global Control via Main IMU
Once chord logic is stable, I’ll use the IMU on the main sensor board to implement global system controls. Specifically:
Hand up = music active
Hand down = music muted
Potentiometer = key selection
This will allow performers to shift keys and toggle sound.
Phase 7: Recording and DAW Control
The final prototyping phase will involve building a simple recording control system:
A Python terminal input or interface will toggle the DAW into record-ready mode
2. A hardware button on the sensor board will start and stop recording
This will support the idea that the device can serve not just as an instrument, but as a compositional capture tool.
References and Tools
Article: https://medium.com/@stevehiehn/how-to-generate-music-with-python-the-basics-62e8ea9b99a5
Mingus: https://bspaans.github.io/python-mingus/
Mido: https://mido.readthedocs.io/en/latest/
Adafruit LSM6DS3TR-C IMU: https://www.adafruit.com/product/4503
Adafruit PCA9548 I2C Multiplexer: https://www.adafruit.com/product/5626
Play Test #1 and Progress Check-In (April 24, 2025)
Current Prototype Status
At this stage, I have a working Python script that:
Takes terminal input (key, chord names),
Converts this input into MIDI-compatible chords using music theory rules,
Sends the MIDI output to a connected DAW.
In parallel, I’ve been prototyping the hardware portion of the wearable interface. I’ve successfully:
Received accelerometer and gyroscope data from a sensor board,
Initiated setup with the PCA9548 I2C multiplexer to connect multiple IMUs.
However, I’m currently troubleshooting the multiplexer channel communication and evaluating whether IMUs are the right choice over flex sensors. This is a pivotal decision point, as the direction I choose now will determine the rest of the build.
Assumptions I’m Testing in This Playtest
Do users understand what action they need to take to “play” the wearable interface?
Is the musical output clear enough that users feel a connection between movement and sound?
Does the gesture mapping feel intuitive or need refinement (is up/down motion enough)?
Questions for Testers
What movements did you instinctively try with the wearable? Did any feel “musical” to you?
Did the sound output feel responsive and meaningful to your motion?
What feedback would help you feel more confident using this instrument in real time?
How critical is incorporating hand gestures to the idea of a midi instrument?
Any other “on the go” ways of generating music?
If you don’t read music, how helpful and generative would you find this process?
Where I Am / What’s Next
Hardware:
Sensor board functional
Single IMU working (in progress)
Multiplexer connection to multiple IMUs
Software:
Python MIDI chord converter functional
Real-time gesture to chord integration
Physical Build:
Need to prototype a secure, wearable mount (elastic/Velcro alternatives)
Translation Layer:
Map accelerometer/gyroscope (or flex) data to chordal input logic
Timeline: Daily Objectives
Thursday (Today)
Finalize IMU multiplexer channel handling
and/or order flex sensors
Create simple test to display each IMU’s data separately
Friday
Compare flex sensor vs. IMU side-by-side on one finger
Decide: commit to IMU or flex path
Saturday
Build a basic wearable for one hand (secure and test for comfort)
Start writing parsing logic for gesture recognition to chord
Sunday
Hook up gesture parsing to MIDI script
Conduct first integrated test: move hand, hear chord
Monday
Conduct internal playtest with updated prototype
Refine output clarity, adjust mappings
Tuesday
Polish user interface and feedback experience
Post reflections and gather additional feedback
Wednesday
Final refinements, stress test all components
Prepare presentation materials
