Consumer IC Advances -> Altering algorithms to create '3D' sound

Altering algorithms to create '3D' sound
By Denis Labrecque, Director of Marketing, Analog Devices Inc., Audio Rendering Technology Center, Mountain View, Calif., EE Times
December 18, 2001 (1:08 p.m. EST)
URL: http://www.eetimes.com/story/OEG20011218S0047

Anyone who has ever played a game recently is well aware of the dramatic improvements made in visual realism. On the flip side, the lag in audio realism has hindered the potential for true game-playing immersion. With consumers demanding better audio performance from their PCs and game consoles and with 5.1 channel "surround-sound" and other audio standards such as the AC97 codec being embraced by the PC market, the infrastructure for robust audio performance is in place. By perfecting algorithmic technology used to render audio at a hyper-realistic level, we have now reached a point at which software designers can explore new functionality beyond that offered by current sound card implementations.

MIDI wavetable playback and 3-D audio positioning are now commonly produced without the need for a sound card. However, while the bar has been raised for effects like 3-D positioning, not much has been done about the actual generation of the sounds placed in these environments. Sounds in the real world are constantly changing. As engines rev up through their gears, they whine and rumble. As characters take footsteps, the surfaces change, and sounds made by people running are different from those made by individuals who are walking. As crowds react to an event, they boo and cheer. But even with today's state-of-the-art hardware, these kinds of sounds are difficult and inefficient to represent realistically with simple static-wave playback and repetitive loops.

Sound production extension (SPX) technologies — an advanced audio-rendering solution developed by Analog Devices and based on algorithmic principles — alleviate the repetitive nature of sound effects by altering the way that they are created. Rather than simply reproducing individual sounds, SPX technologies allow game developers to utilize a series of audio behaviors that accurately reflect how real sounds are produced and perceived. The SPX sound descriptions are stored as sound algorit hms, which are files that contain only the compact instructions that are needed to create and render sounds on demand.

Basically, three types of sound effects are used in games: continuous effects, one-shot/singular effects and ambient backgrounds. Due to the fact that high-quality recorded sound requires a large amount of storage (10 Mbytes/minute for a mono file), these effects have relied historically on either repetitive loops of small wave files or on the reuse of the same short file every time the sound is needed. But continuous effects, such as engine sounds in racing games, are often poorly represented by looping and pitch shifting a recording of an engine. This method cannot accurately reproduce the dynamics of how a real engine works.

In addition, sounds in real life can be quite complex: a car crash, for example, is not a single sound, but is a composite of screeching tires, breaking glass, scraping metal, etc., all of which sound different each time they occur. In most games, the same c ar crash "sound" is repeated every time your vehicle hits the wall. As any game player knows, this can quickly make the sound of most games very boring. SPX Technologies curb this repetition by providing several methods in which new sounds are created in real time, in response to the players' actions. In addition, it provide other techniques that achieve similar results by combining a series of actual recordings with advanced algorithmic looping and crossfading techniques, providing "signature" sounds of specific vehicles.

One method used to add audio realism is physical modeling technology, which mathematically recreates the actual physics of a sound. The technology was originally created at Stanford University Center for Computer Research in Musical Acoustics and was further developed at Staccato Systems, which was acquired by Analog Devices.

SPX audio algorithms actually calculate an engine sound based upon the physics of how a real sound is produced. A mathematical model of the number of cyl inders, resonant chamber sizes, materials, densities and other criteria are combined to produce a sound that is constantly varying and responding to the actions of the game; changing gears, rapid acceleration and even engine damage are very accurately represented by this method.

Event modeling can also produce non-repetitive game sounds. These algorithms render sounds based upon the random, yet controlled, triggering of the individual dynamic elements of a sound to produce complex events, such as a car crash.

Event modeling is another way of capturing the behavior of a higher-level physical process. It is a synthesis method which models large-scale sound events by encapsulating collections of smaller, simpler sound events and their interactions. For example, a large-scale event may be a car crashing into a wall; and the component sound events may be individual crunch sounds, the sounds of various objects falling on the ground, various scrape sounds, etc. The intention is to be able to generate larg e-scale events that can be repeated many times without sounding repetitive, or to generate rather long "events" which don't sound like loops. It is also possible to place interactivity into the algorithm by varying the parameters of the algorithm.

The results are highly realistic, with a degree of flexibility and dynamism unavailable via any other method. Footsteps, for example, are actually elaborate sound events in real life. They depend on a number of variables: the surface (grass, concrete, wood floor, metal stairs); the type of shoe (boots, cleats, bare feet); and pace and style of walking. SPX allow this complexity to be reduced to a database of small recordings controlled by an algorithm that represents the behavior of footsteps in a number of environments.

Ambient backgrounds such as forest sounds and train stations are composed of both continuous sounds and added individual events. These effects are created by using a combination of loops and events, with SPX Technologies altering many pa rameters in real-time, such as density, duration, order and frequency. The algorithms also provide a degree of randomness that is necessary to make the sounds unpredictable and hence, more believable.

Algorithmic synthesis methods are becoming more important in sound reproduction for games and music. One attractive feature of the general algorithmic synthesis programming environment is the ability to combine various kinds of synthesis and processing effects in creative ways. Wavetable synthesis can be very cost-effective for creating certain kinds of sounds, though it may be less lively than the more costly physical model of the same sound. However, driving a wavetable synthesis algorithm with a physical process model or mixing a few recorded sound files into a physically modeled game sound environment can result in a synergistic hybrid of synthesis algorithms that are greater than the sum of their parts.

The recorded sound files can add realism to a flexible but synthetic-sounding algorithm, where as algorithmic synthesis can add flexibility, interactivity and imagination to an accurate but repetitive-sounding wavetable sound. The combination of wavetable, physical modeling, DSP and any of the other algorithmic synthesis techniques within a single sound environment can lead to a flexible, interactive, larger-than-life listening experience.

To incorporate these new SPX technologies into games, Analog Devices' Audio Rendering Technology Center is developing a series of tools for PC and console game developers. The SoundMAX SPX Game Audio Toolkit provides applications that allow developers to select and control the algorithms and sound files that make up complex audio events. The designer can then use these audio behaviors via a simple-to-use application programming interface that lets developers integrate game sounds that are controlled by both player input and game parameters.

Shipping on a number of major computer systems, SoundMAX SPX Technologies can scale to accommodate the differences in a variety of systemfrom PC's to console systems such as the Xbox and PlayStation 2. Games that have begun to incorporate SoundMAX include MotoRacer 3 from Infogrames and the upcoming PlanetSide from Sony Online Entertainment.

See related chart