Model Based Sonification

Source identification

source characteristics

characterizing sound by rhythm, harmony, melody, patterns, etc.

"the conscious use of all listening skills to distinguish and analyze an object under investigation"

"task of determining the contents of an opaque box by shaking it, or the task of diagnosing a malfunctioning car engine from the sounds it makes"

Haptic feedback loop

Friction and radiation cause energy loss and cause the system to tend to equilibrium

Physical systems often produce sound because our interaction actively perturbs their state

"Interactions are followed by physical/acoustic reactions which then lead to auditory perceptions and their cognitive interpretation"

Most interactions should be followed by sound

The sound producing algorithms should be independent of the actual data values

Low latency

Real time

Sonifications should depend on a subtle level of interaction

i.e. there has to be sensitivity to small changes

Sonifications should be "non-trivial"

Complex, many layers

how data define the configuration of a dynamic system with internal degrees of freedom

Taking high-dimensional data and projecting it to a physical space that changes in time

Converting data space into model space

The configuration of the model right after setup

Put the model in position ready for user interaction

Can be used to avoid loud noises or large movements towards equilibrium

Define the dynamic behavior of the model in time

Kinetic and potential energy

Dissipation and friction

Most of these equations will allow for arbitrary parametrization

Excitation feeds energy into the system

Because the system tends to equilibrium it requires excitation to produce sound

Excitation can be modeled as an external force applied to the system

The force can depend on the state of controller interfaces

Types

Excitation parameters can be made to affect model parameters

Variables from the model used for sonification

There can be a one-to-one mapping

Or multiple variables can be condensed into fewer

issues related to location, orientation or distance between sound sources (link-variables) and the user/listener

sonification model

Real-time interaction is of paramount importance

So corners need to be cut

e.g. use a lower sampling rate, then interpolate

Important to make sure that important subtleties are not thrown away

Each data point is considered a mass in a spring-mass system

The user excites the model, and as the waves traverse the system, they displace the individual masses

sonification.de/handbook/index.php/chapters/chapter16/#S16.2

The movement of the user excites the model directly

There is no excitation unless the user's movement intersects a mass spring system

Principal Curve is a statistical/machine learning technique that computes a smooth path

Data is serialized along a path that can be treated as a time series

Or there can be user interaction in the navigation of that path

Example

Data points "crystallize" following crystillization behaviors

Excitation consists on setting a "crystallization" nuclues

"The metaphor is that the inclusion of a molecule sets free some energy which contributes to the overall vibration energy of the growing data crystal."

"The crystal’s modes of oscillation are not defined in analogy to physics, but instead use the covariance matrix of the data set at each growth step as follows: the eigenvalues determine the harmonic series while the overall variance determines the size and thereby the fundamental frequency of the sound."

"During growth thereby the pitch drops whereas the brightness signature modulates."

Data is interpreted as fixed "masses" in space

The data samples contribute to an overall "gravitational" or "electromagnetic" potential function

Additional particles are made to travel through the data following the laws of mechanics

An additional friction term can be added to ensure particles come to a stop after some time

Collisions are excluded

The excitation consists of adding moving particles to the system

The sonification can be done from the sum of the energy of the particles

The link-variables are the instantaneous particle energies

"The primary analysis task of the model is tomake perceptible the homogeneity and clustering shape of high-dimensional data. The structure can be understood from stable sonic pitch plateaus and noisy patterns during the transitions between these modes"

A sort of self-organizing neural network

Energy can be injected into the neural network.

This energy will spread across the network

The energy of each neuron contributes to the sonification

When a user excites the system, local properties, as well as distant properties, which are heard later

The model can be explained in terms of metaphors, which allow the user to understand the information sonification provides

automatic concept formation

"For cluster analysis, the GNG sonification model, the particle trajectory sonification model, the data sonogram model and the tangible data scanning offer basic tools."

"For understanding the topology and intrinsic data dimensionality, the GNG sonification model and the data crystallization sonification model can be used"

"For understanding multi-scale structures, the growth process sonification of the GNG sonification models, and the particle trajec- tory sonification model (while controlling the bandwidth parameter σ) can be used"

"For understanding the separability of classes in classification problems, the data sonograms with class-entropy-based spring stiffness may be used."

changes in the data do not only change the model setup, but also provide some excitation

mixed-reality applications that support human activity and provide an ambient information display