Very Nervous System
David Rokeby has been making new media artwork since the early 1980s. One of his well known pieces "Very Nervous System" was directly responsible for causing me to dive down the rabbit hole into interactive technology. Here is a video of it in action:
The gallery onlooker is intended to stand in front of a camera and dance around. The result is a beautiful interactive musical composition that pays homage to musical styles found in traditional eastern Asian music. Although the technology and interface has changed many times over the years the basic concept has remained the same.
The image is reduced into a monochromatic grid by a given number of pixels. The amount of light present in each pixel and the change in light that occurs over time is manipulated and mapped to a series of messages that control a musical composition. In the early stages of the installation, 64 individual photocells housed behind a makeshift lens were connected to an Apple II computer using a variety of hardware interfacing. The computer was used to generate MIDI information which was sent to a series of synthesizers to generate the audio for the installation. The complexity of the interaction between how the hardware, software, and synthesis interact with one another can be found in the following article:
Creating Interactive Dance with the Very Nervous System - Todd Winkler
I tried to mimic similar techniques with Max/MSP in the video above. The patch is looking for overall brightness of a specific hue in each square. Each square outputs data that behaves in a similar manner to a photocell. Normally, a photocell measures overall brightness. In the patch only a specific hue's overall brightness was being utilized, in this case the color red. Although the entire thing was done in software and is in fact not an actual photocell, the output of each pixel could be thought of in the abstract as a fluctuating analog signal like a control voltage on a synthesizer. Much like a modular synthesizer, if you were to lock this control voltage to a scale by using a quantizer, you would be able to get musical intervals out of the interaction. This patch does exactly that and assigns those quantized notes to the frequency of individual sine waves for each square. In this particular example the key chosen was A minor pentatonic. Some squares are offset from others harmonically by octaves and fifths to create a more interesting composition.
David Rokeby currently resides in Toronto, ON and is still actively creating awesome artwork. Very Nervous System is still being exhibited.