InterMedia Patterns

Alan Merriam, in the Anthropology of Music, devotes a chapter to “Synesthesia and Intersense Modalities,” where he offers a number of terms to express various types of intermodal and intermedial experiences.

He briefly discusses synesthesia, defining it along familiar lines such as “…the experience of an associated sensation when another sense is stimulated” (Carl Seashore). This is the classical neurological synesthesia.

He also discusses a neurorological phenomenon akin to synesthesia where stimulus in one sense affects the acuity of perception in another. This he refers to as intersense stimulation. It appears to be a common physiological phenomenon that has been statistically evaluated. How do smells affect your hearing? How does light affect your sense of taste?

Intersense Transfer involves the perception of sensory qualities in different modalities as similar. “Rough” and “smooth” sounds and surfaces, for example. It’s partly a cultural construct, but probably not entirely: language serves multiple senses, but language evolves with experience.

Intersense Modalities are culturally constructed associations of sensory modalities. For example: association of colors with cardinal directions in many cultures. Artistic systems (Scriabin, Rimsky) fall into this category–and I would suggest that the question of whether Rimsky or Scriabin or Rimbaud were synesthetes is largely beside the point. They constructed systems that served their artistic purpose in a cultural moment when such correspondences were part of avant-garde practices. In Rimbaud’s case, there is the Symbolist tradition starting with Baudelaire. Rimsky and Scriabin were in contact with the Theosophists and any number of theories of unification of the arts. These influences made it possible to develop a cross-modal artistic language that would have symbolic potential.

Sensory Multimodality is a cognitive principal that asserts that the senses evolve as an integrated source of information (consider the recognition of food). This principal is evoked in the cultural critiques accompanying some multi-modal art. It is suggested that the senses become separated by education, routine, exposure to media. Multimodal art then restores an “original unity of the senses.” (Actually, count me among those who think this point of view has some considerable weight).

We might also consider how gesture functions within sensory integration. Studies of kinesthesis (the sense of the body’s position, movement, and gesture, also referred to as proprioception) apparently reveal that kinesthetic sensory qualities of a gesture are reproduced in our imagination when we perceive the sound or the mark that the gesture creates, without the gesture actually being present.

I think we could agree that “gesture” is a cross-modal category: intensity over time, no matter the modality by which we experience it. Gesture can be informal (spontaneous) or formal (measured and systematized). Gesture can be reversed in time, or inverted in intensity–then we have counterpoint. Because it can be mapped onto different sensory modalities, gesture supports correspondences and mappings from one sense to another: loud:soft :: bright:dim, and many others.

In light of the above categories, I would assert that when artists combine artforms intended for different senses, they are not necessarily doing so in reference to synesthesia. Correspondences are constructed between events in different sensory modalities because we have a wide range of experience in which the senses interoperate. Even the specific category of “synesthetic art” may spring from a cultural tradition that originates in other cross-modal experiences and later becomes identified with synesthesia because of the symbolic weight accorded to multisensorial experiences. I think this is clearly the case with the Baudelairean tradition of “correspondances.” I can think of very little synesthetic art that is just intended to simulate a synesthetic state–even when that is its avowed purpose, it nearly always freights the work with other communications, with symbolic meaning that recreated experience alone does not carry.

In any case, the idea of synesthesia has been more useful to the arts (in the form of poetic fusion, metaphor, and cross-modal structure) than the actual phenomenon. By this, I mean in no way to suggest that synesthesia is not a fascinating phenomenon from which we can learn a great deal, only that it is one province of a wide realm of intermedial and intermodal experiences and cultural representations. I think it is particularly important to place artistic endeavors within this wider and more richly varied context. If we want to explore multimodal art, we might as well draw on the entire context, both in our creative and our analytical approaches, and not attempt to stretch the bounds of “synesthesia” too far, when there are other descriptive terms.

When mapping from one sensory channel to another operates in real-time with a relatively high temporal or spatial resolution, we can construct an interface by providing a point of interaction–e.g., mouse movement translates into cursor movement, pressure on a pad translates to amplitude control of a sound, etc. When the mapping is slower or not interactive but provides an “objective” rendering of information, where scalar values are conserved across the mapping, we a probably looking at a visualization–e.g., a signal is rendered by an FFT as a histogram of frequencies, weather data are mapped into audible frequencies, etc. The visualization may be delivered as an instrument, such as an oscilloscope or sonar display. If the mappings are at least partly symbolic, or involve subjective renderings (in which we include aesthetic choices), we are probably dealing with intermedia art. Intermedia patterns potentially have something to offer to all three disciplines.

I do not as yet know much about this field and the WikiPedia entry on it is rather slim. Much of the work in the field seems to originate with one person, Yingxu Wang at the University of Calgary. Those disclaimers apart, it seems worthy of interest on two accounts. First, it suggests that Information is a category co-equal with Matter and Energy, a point of view very much of our time when material objects appear charged with informational patterns (i.e., a state of virtuality, as N. Katherine Hayles defines it). Second, it proposes a model for computation apparently influenced by theories of embodied or distributed consciousness, and thus for constructing a new generation of computational devices (entities?) that interact with the world.

It also apparently offers a mathematical notation for modeling its theoretical discourse. To be sure, cybernetics and general semantics have made similar claims about modeling consciousness, with variable success. Wang’s articles are found in the IEEE Proceedings, and should be available online through Project MUSE and other journal databases to institutional subscribers (check your university library).

I have put the materials on this site under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 Unported License. This seems the best way of both sharing and protecting the text. The icon at the top of the sidebar and the footer text both indicate the licensing.

Novelists and psychologists share an interest in the way we think, argues Charles Fernyhough, but writers must do more to keep up with science.

Steadily, in vivid colour, the brain gives up its secrets. Hardly a week goes by without the news that another elusive human quality - the capacity to understand sarcasm, to give one recent example, or to judge another person’s trustworthiness - has been traced to a side-alley of neuroanatomy. Away from the fMRI scanner, psychologists and cognitive scientists - those who study the software that runs on the brain’s hardware - have made great progress in the modelling of human thought processes. Although the big problems of consciousness and free will show little sign of yielding to scientific analysis just yet, at least some of the mysteries of mind and brain are close to being accounted for by objective, testable theories. Link

Charles Fernyhough
Saturday October 15, 2005
The Guardian

I upgraded the blog software to Wordpress 2.3.2, an upgrade that included a security patch and so seemed rather necessary. In the process, I had to change the mechanism for loading TinyMCE, the HTML visual editor. Previously, I was able to load the editor in a compressed (zipped) version. That seems to be broken, so I am loading the uncompressed version. If you have any problems with editing, clear your cache, quit your browser, launch the browser again, and try editing again. If it doesn’t work, contact me. Versions prior to 2.3.1 had the same problem. I’m not sure what is going on.

Compression and Clashes

In an elaborate typology of networks, four kinds stand out on a continuum of complexity: simple, mirror, single-scope, and double-scope. At the high end of the continuum of blending complexity, double-scope networks blend inputs with different (and often clashing) organizing frames to produce creative emergent frame structure in a blended space. Double-scope blending is what we typically find in scientific, artistic, and literary discoveries and inventions. Indeed, double-scope creativity is perhaps the most striking characteristic of our species.

from The Way We Think, Fauconnier and Turner

Conceptual Mapping

Semantic and formal mapping
Linguistic and structural metaphor
Integration Networks

Single Stream

• gradient
• cycle
• container, border, edge (inside/outside)

Multiple Stream

• counterpoint
  • parallel motion
  • contrary motion
  • divergent/convergent motion
  • retrograde motion

• symmetry transforms
  • affine: reflection, rotation, scale, skew
  • non-uniform transforms: warp, wave, whirl
• strategies
  • juxtaposition
  • intersection
  • complementarity

Collections and Networks

• list
• ordered list, sequence
• tree, taxonomy
• network, graph

Selection Rules

sampling
fragmentation
affinity
disparity

From P. Adams Sitney’s commentary on Maya Deren’s “Meshes of the Afternoon”:

Only a painstaking analysis of “Meshes” reveals its efficiency: each time we see a woman’s hand—with the flower, the key, or the knife—each time an object is transformed into something else, each time she looks out the window or enters the house, some very slight nuance is added. Every image, every transition rhymes with another and is ultimately necessary either to creating the illusion of continuity or to establishing the ambiguity between dream and reality.
[P. Adams Sitney, A Psychological Analysis of Meshes.]

I was struck by the combination of formal and semantic techniques suggested by this passage, quoted in the Criterion DVD edition of Maya Deren’s films. The formal combinations of elements in their relationships (”rhyming”) elicit an illusion of continuity and meaning. The “illusion of continuity” is a one aspect of studies in cognition, particularly the continuity of the visual field, but we might well ask how the illusion of personal continuity rises out of the combinatorial engine of the mind—give this problem a turn, and perhaps we also confront the artist’s difficulty in creating continuity out of interrelated elements. How do disparate elements become parts of a whole? Does meaning arise out of that act of synthesis?

Notes by Jack Ox

Time = Space Metaphor

Time = Space is an integrated conceptual network, and in this capacity, it has been written about extensively as a complex, interactive system. [Rethinking Metaphor, Fauconnier & Turner ‘06] They point out one power of an integration network is the ability to develop emergent structure based on pre-existing conceptual structures while also achieving compression of these ideas. Time = Space was one of the first metaphors Ox used for mapping music into a visual language. In the domain of music time functions as syntax. In the domain of Visual Image it is space that functions as the syntax.

Space can show how long events last, and their relationships to other events. It also is useful for mapping differences in pitch, showing a higher placement for higher pitches. So, higher and lower, elements of pitch sequences, can be understood through the placement of objects in space which are higher and lower than one another. These relationships are relative because one has to assign the first value arbitrarily in order to begin.

Movement Metaphors and Structural Metaphors

Space can also be thought of as places along a graduated path, e.g. a scale ranging from fully saturated colors to white is a graduated series that moves through various levels on the way to and from the endpoints. It is the movement along this line that defines space, and this reflects changing status through time.

When Ox visualizes changes in status, i.e. in music the change from one musical key to another, an array of values in the visual world is found that can be moved through with the same level of detail that exists in the source domain. Some examples are the tracking of relative states of consonance versus dissonance and key changes in harmonically based musical structures. Changes in consonance and dissonance are mapped onto a grey scale, which includes changing percentage formulas for the combination of complimentary colors in the final color. This system works when one must map the steps between two different values. For mapping movement through an ordered cycle of values in one domain, then an equivalent cycle of values must be utilized in the visual world, e.g. movement through different hues. These two systems of movement can be combined and then come together into an emergent state, the sum of the two different values is greater than the parts.

Because the proto-typical color wheel has 12 different colors it is logical to map that circle to the circle of fifths, which also has twelve different keys.

Another metaphoric movement pattern is the change between different levels of saturated and unsaturated states. This can be mapped to changes of pitch- as in higher pitch equals lighter color and lower pitch equals a more saturated color. Saturation can also be mapped to changes in amplitude.

Container Metaphor

The orchestra is a container for the families of instruments, each of which contains individual instruments. The Container is one of the most broadly used metaphors, although Lakoff and Johnson (1980) treat the Container as an image schema, variously called “in-out schemata” or “in-out orientation.” In the First Organ Stop of the Virtual Color Organ™ Ox maps various families of instruments to desert landscape units, each of which bears perceived structural similarities to the group of instruments being represented. The complete landscape scheme is therefore a metaphor for the entire orchestra.

The Symphony is also a container in that it contains the parts of its musical form, i.e. four movements. When visualizing the Eighth Symphony of Anton Bruckner, Ox took architectural imagery from the Baroque monastery where Bruckner studied, played the organ, and is buried, utilizing discreet units of the building complex in mapping the larger musical units, called movements. These architectural units also functioned as Structural Metaphors where architectural details reflected pitch patterns contained in the music. St. Florian’s Monastery functioned as both the container of the symphony mapping to both the larger structure of the music and also details found in musical themes.

Subjective experience versus Objective experience:

When visualizing information in the Arts one is often dealing with relative values instead of absolute values. The reverse is true in Scientific Visualization. When NASA is visualizing the placement of stars and planets in space, they are dealing with absolute placement of points in space.

In visualizing information from the Arts the aspect of style must be taken into account. If the source exists within a style, this style also exits within a particular range of time. Elements may be used in the mapping from another media domain within the same time period. For instance, Ox used Baroque architecture when depicting the Baroque musical construction of melodic counter-point. Other aspects of style, i.e. the notion of Romantic in the late 19th C., have been called up by using dramatic landscapes that have been rendered in a dramatic, Romantic style.

Timbral Analysis/Color Metaphor

Timbre is everything but pitch, amplitude and spatial perception. It is also often referred to as the color of sound.

Timbre, like pitch, is tied to human perception, and is therefore inherently subjective in nature. A psychoacoustic property, scales have been developed to describe timbre, and they use adjectives such as dull/sharp, compact/scattered, bright/dull, and full/hollow. These terms are subjective ways to refer to sound, although they do convey relative, rather than absolute, meaning.

Timbre can be measured scientifically by analyzing the formants. Vowel sounds, the producer of timbre in human speech and singing, can also be read through analysis of their formants. Human voices can vary widely in range and still exhibit the same formant frequency ratios in vowel production. Therefore pitch range is not a part of timbre.

Vowels can also be analyzed by looking at how and where they are produced in the vocal tract. This is the system that Ox has applied to timbre in vocals. However, this data can be extracted real time by using the formant measuring system. Then, one could also use the same formant system for real-time measurement of timbre in sound produced by musical instruments.

Ox relied on her ears and hearing for analysis of timbre and mapped the results to a complex system of timbre she created in consultation with a composer and orchestrater. An interesting experiment could be set up to compare the results of these two different methods of data analysis.