Hi.

Welcome to my blog. I document my adventures in travel, style, and food. Hope you have a nice stay!

The Wearable Pokey Slappy Four-Handed Sequencer

The Wearable Pokey Slappy Four-Handed Sequencer

Originally posted on Foundry.

The Wearable Pokey Slappy Four-Handed Sequencer is a wearable electronic musical instrument that is played by two people and sound-generating software. Both players wear a pair of gloves. The gloves have patches of conductive fabric on the fingertips, the base of the palm, and on the knuckles (where you might give someone a dap). 

 The gloves on display at the  University of Texas Visual Arts Center .

The gloves on display at the University of Texas Visual Arts Center.

Players create music by touching each other’s fingertips. Each time they touch a pair of fingertips together, it adds notes to an invisible step sequencer. 

A typical step sequencer is a simple electronic music composition tool that is typically is laid out in the shape of a grid; the grid can be any size. The vertical axis of the grid represents pitches and the horizontal axis represents beats. The sequencer plays through one beat-per-column, and then repeats. When you touch a square, it adds it to the sequence based on its location. You don't need any musical experience to compose with a sequencer because the notes are always locked into place within that key structure and beat structure.

Here's an example of a typical step sequencer - 

In the case of the gloves, there is no visual step-sequencer. The gloves work like this: Players add notes to a musical sequencer by touching fingertips with a partner. The eight fingers of the gloves correspond to an eight-note musical scale and the thumbs shift the scale up or down an octave. Players can choose from four MIDI-based timbres, and can create more complex arrangements by layering up to four simultaneous tracks.

Why I made them

By trade and in my heart I’m a designer. I began the project because I was curious to learn about a few things:

Embodied Interaction: It is important for designers to learn to imagine and prototype interactions that go beyond the screen. I wanted to challenge myself to envision and build an interface that focused on real, tangible interaction. UC Irvine Paul Dourish describe embodied interaction as “the property of being manifest in and of the every-day world. Embodiment constitutes the transition from the realm of ideas to the realm of everyday experience.”

Mutually Reliant Interaction: In other words, a player must engage with and touch the other person in order for this musical instrument to work. I was also extremely curious about what sort of dynamics would happen as a result of this interaction. Would people feel uncomfortable? Would they create music together, or would one person dominate? In some ways, this project became sort of a social experiment.

The Balance of Freedom and Constraints: I wanted to learn how to design a tool that was open enough to allow for creativity and expression but had just the right amount of constraints to help the players create something that sounds interesting no matter what.

Non-Screen Based Feedback: The feedback players receive while using the gloves is a combination of audio feedback and the feeling of your partners’ fingertips. While there is a visual interface for the software (seen below) that allows you to set the initial parameters (musical key, speed, etc), once the players begin there is no need for visual feedback. 

 The sequencer is controlled with the gloves and written in  Max MSP . It can also be controlled directly through the computer keyboard.

The sequencer is controlled with the gloves and written in Max MSP. It can also be controlled directly through the computer keyboard.

n9kvvhowikgyvkbjr4au.jpg

How they work

  • The gloves have conductive fabric on the fingertips, the base of the palm, and on the back of the knuckles right where you touch hands when you give someone a quick dap (fist bump).
  • The fingertips from pinky to index finger on both hands are the musical intervals of Do, Re, Mi, Fa, So, La, Ti, Do in whatever key you choose.
  • The left thumb brings you down an octave and the right thumbs bring you up one.
  • To begin and end a sequence you bump right palms together.
  • By touching left palms, you change the midi instrument and the track.
  • There are four available tracks that you can layer or sequence with expansive complexity. 

When the two players touch fingertips together it completes a circuit, which sends a signal to the software to trigger a note. This doesn't work when you touch your own fingertips together; that will not complete the circuit. It only works when you touch fingertips with the other person.

No musical experience necessary because the key and beat structure is locked into place for you.

People had different ways of exploring the gloves. Some people dove in and started banging their fingers together with abandon. After a few seconds of this they would stop to listen to how the software grabbed the notes and placed them into a sequence. Others were more intentional and crafty, trying to construct something a little more specific. These two extremes represent a spectrum, and most people ended up playing at multiple places along that spectrum during their experimentation.

How I made them

I wanted to get to the point where people were actually using the gloves as quickly as possible. The gloves were purchased from a thrift store. I used hot glue to fix the conductive fabric and thread to each glove. Although this produced a less attractive and less durable result than sewing, I was able to produce a workable product in an extremely short amount of time and with zero pinpricks. The workable version of the gloves took about one day to create. 

In order to get this running as soon as possible, I cheated a little: The prototype used a Makey Makey board to turn each fingertip into a switch. The simplicity of the Makey Makey allowed me to cheat and learn about the interactions before I had a technically complete design. As I mentioned earlier, the gloves work by completing a circuit. In reality, only one set of gloves is connected to the notes. The other set is a ground that can only complete the circuit.

I am an extremely amateur programmer. I wrote the software in Max MSP , a visual programming language used primarily for interactive music and video. Max is great for visual artists and designers who’d like to doodle with code because it lays out your code in front of you as a visual system. This allows for an experimental programming experience that feels more like bricolagethan the systematic problem solving typical of most seasoned programmers.

 A snippet of the underlying Max patch.

A snippet of the underlying Max patch.

What I learned

I was and still am pleasantly surprised by how much fun people have with these gloves. Some people just get into it and start smacking their fingers against the other person’s gloves and then they listen to it play back and listen to the unique combinations. Of course, they probably couldn’t reproduce the sounds if they tried, so it’s more an experiment in how well the gloves turn even the most haphazard approach into something that sounds reasonably interesting.

Before I tested the gloves, I was curious about the learning curve. Since the players have no visual reference, I wondered if the controls would be intuitive or confusing, comfortable or awkward. Patterns emerged. Initially many players looked to me for direction: “Am I doing it right?” I assured them that there is no “right,” and to just play. 

I was pleased and surprised at how quickly most people picked it up. In one round of experiments, there was a very clear “ah-ha” moment when the players shifted from touching fingers and being surprised at what happened to suddenly achieving the beginnings of proficiency. All of a sudden one of the players started moving deliberately, skipping through octaves with her thumb, changing the track with her hand, and creating something with more intention than she had previously.

I also learned about simplifying functionality along the way. There were a couple of capabilities that were a part of the un-tested prototype that I learned were unnecessary and confusing. I removed them.

This project not only taught me about creating interactive objects, it also taught me about the social dynamics associated with a collaborative creative act. A friend jokingly described this project as a “social hierarchy identifier,” as almost every time one of the players emerged as the dominant of the pair. 

What's next

I'd love to find a collaborator and work together to create a new version that is simpler, more durable, and more expressive than the first prototype. 

During my MFA research at the University of Texas, I was able to spend some time with Rachel, a music therapist who worked with kids on the Autism spectrum. Her work is fascinating and moving. She maps developmental goals such as verbal communication, memory and sequencing, and making requests onto musical elements such as rhythm, melody, harmony, volume dynamics, and song structure.

On seeing this first prototype, she explained to me how a tool like this could be used to help kids work on developmental skills like joint attentiontouch desensitization, and cooperation. 

I think there are entire universes of opportunity for creative technologists to build playful and powerful tools to help people like Rachel.




Electric Singing Amoebae from Outer Space

Electric Singing Amoebae from Outer Space

"Childhood" At Creative Mornings ATX

"Childhood" At Creative Mornings ATX