The set of objects is expanding while we are exploring the application of
the objects to implement sound sculpting [3]. Figure 3
illustrates the
approach we are taking in mapping human movement parameters to sound
parameters. It is our strategy to use shape as a means to relate hand movements
to sound variations. Hand movements, shape and sound are all multidimensionally
parameterized. We exploit the intuitive relations between
shape (of physical objects) and timbre as well as shape and manipulation
for the design of a sound editing environment where the user can change the
sound by applying shape operations to a virtual object using a shape processor.
Shape features are subsequently computed and mapped to sound parameters.
Hand features that are computed from the acquired hand shape, hand position
and hand orientation data using the above set of objects are mapped to shape
parameters of a similar abstraction level. If a shape parameter is
specified at a different abstraction level than the parameters of
the shape processor, it will need to be translated to the parameter space of
the shape processor. A shape processor is an algorithm that computes surface
data points from shape parameter values. The simplest shape processor is the
identity, i.e. it takes surface data points as shape parameters and outputs
the same surface data points. If the hand feature computation and shape parameter
translation are also taken as the identity, the positions that represent
the hand will also specify the virtual object surface or in other
words, the hands are always "touching" the virtual object.
A shape feature is computed from surface data points and represents an
abstraction of the shape. For shape feature computation we have developed a
number of objects. The object
plane, discussed also above, computes the plane normal vector from a list of
positions.
Another type of shape feature computation involves superquadrics, a mathematical
method to describe a wide variety of shapes with two parameters specifically
related to shape (vertical and horizontal "squareness") and 9 others for size,
orientation and position of the virtual object. A superquadric surface is
defined by the 3-D vector
Based on [6] we have programmed an superquadric object that fits a superquadric shape to a set of positions. As the fitting process is iterative, it is computationally expensive and as yet too slow (order of 100 ms) for real-time control of timbre. A simpler approach is implemented in a feature computation object ellipsoid which fits only an ellipsoid to the list of positions and computes within real-time. Currently our fitting algorithms do not take into account that the virtual object should be bounded by the hands, so that the virtual object intersects with the hands. Thus, we need to add more constraints. We are further studying 2-D and 3-D Fourier transforms for use as shape features [1]. Figure 4 shows a typical example Max project that allows a user to control FM synthesis parameters using hand movement primitives.
Figure 3: Functional diagram of hand movement to sound mapping. Shape is used as
a means to relate hand movement features to sound features. The diagram intends to
illustrate mapping at a similar abstraction level of hand movement parameters
to shape operators and of shape features to sound parameters. The example shows
mapping at the feature abstraction level, but mapping at other levels is
equally possible. While the hand feature and shape feature concepts appear in
this diagram at the highest level of abstraction we are planning to develop
other computations that use the features to derive higher
abstractions of hand movement and shape.
Figure 4: Typical example of a Max project using the new Max/FTS
objects to compute a fit of the superquadric shape parameters to the list of
positions as computed by the geoHand objects. The superquadric
shape parameters or features are subsequently sent as messages to the
client object which sends them to the OpenInventor 3-D graphics server. The
parameters are also mapped to FM synthesis parameters like modulation and
envelope and sent to another Max project for good old FM sound synthesis.
With reference to
figure 3, the character device driver is not visible because it is
a subroutine called by the protocol handlers cyberglove and polhemus.
Also, in this example we have set the hand feature computation,
shape parameter translator and shape processor to the identity operator so that
the list of positions generated the geoHand objects
are taken as the new surface data points as input for the superquadric shape
feature computation.
The sound parameter translator and processor are represented as another
Max/FTS project, not included in the figure.
