Articles‎ > ‎

Synesthesia As Algorithm

posted Jan 30, 2013, 10:27 PM by Ellen Pearlman

Synesthesia As Algorithm

Partitura 001 by Quayola

The term Synesthesia derives from ancient Greek and translates as “together” “sensation.” Artists think it is natural and wonderful gift. Science thinks it is a “neurological condition in which stimulation of one sensory or cognitive pathway leads to automatic, involuntary experiences in a second sensory or cognitive pathway.”

Synesthesia powers development in 3D visualization and concurrent sonic referents. Without that type of relationship there would be no dynamism or compelling reason to view and participate in art. 

Quayola has worked on developing Partitura (which means score) as a custom software in the program vvv to generate realtime graphics that visualize sound. A custom Max4Live patch enhanced the sonic effects. The visuals are set down according to a “spline-driven bones structure” that all other sections refer to including Particles and Meshes. The paramater of all other modules reference are lined to OSC inputs from the audio analyzer  or a MIDI sequence/controller. The output analysis is run through Ableton Live and sent over OSC via an ethernet connection. 

What does all of this this actually mean?

Partitura starts with a horizontal line, just like a musica score. Three classes of abstract elements are created and change according to  sonic inputs using different classes of precise mathematical calculations. 

First there are musical notes refined into wave forms

Then the waveforms get translated into abstract visual mappings

Rendered visual mappings with sound

The visual mappings are rendered into geometric shapes that change according to the music in a beautifully synchronized manner.

In a sense, this harkens back to very early experiments in visualization includingOskar Fischinger as shown below in his 1938 piece “Optical Poem.” That work was all done in stop frame animation painstakingly put together piece by piece with cut outs.

Optical Poem (1938)