MMI722 AUDIO FOR GAMES AND VIRTUAL ENVIRONMENTS

Course Objectives

Computer games and virtual reality systems rely on spatial audio to provide the desired level of realism and immersion. If a player/user is positioned in a virtual room with other sounding objects, the realism of perception, localization of these sound sources, and the perception of the virtual environment requires auditory feedback from the modeled virtual environment. Similarly expressive synthetic audio is necessary to improve the realism of such applications. Sound objects in computer games are typically canned (i.e. pre- recorded) sounds, also known as foley sounds. This approach is useful, but insufficient in that repetition of canned sound effects cause boredom and reduce realism. This course aims to provide students with the necessary background and skills together with an in-depth understanding of virtual and synthetic audio.

The course will first aim to bring all students to the same level on fundamental concepts of digital audio, including, sampling, frequency-domain representations, filtering, and time-frequency analysis. Human auditory perception, including concepts such as critical bands, perception of pitch, timbre, loudness, rhythm, and texture; as well as masking will be covered. Spatial auditory perception will be covered specifically including its aspects relevant to game audio such as binaural hearing, localization of sound sources in 3D space, and distance perception. Concepts related to room acoustics modeling, simulation and auralization will be covered from game audio perspective. Artificial reverberators which are strongly related to perception-based simulation of room acoustics will be covered in detail. Most important existing and emerging spatial audio reproduction techniques will be covered. Subjective evaluation of spatial audio systems and algorithms including experiment design, ITU recommendations, and statistical analysis of results will be discussed. Coding of spatial audio for use in computer games and virtual reality applications will be given with existing MPEG standards. Discussion of sound synthesis will include a brief review of

existing speech, musical instrument, and environmental sound synthesis methods. Special emphasis will be given to expressive synthesis and morphing techniques that allow using the same sound synthesis method to generate perceptually similar but not identical sounds. Finally, existing game audio APIs will be reviewed.

Course Content

Spatial and synthetic audio are key elements of computer games and virtual reality applications where a high level of realism and immersiveness is desired within computational limitations of available hardware. The entire chain of processes from the production of sound to its perception all play part in the success of game audio systems. This course aims to introduce the fundamental concepts of spatial and synthetic audio for computer games and virtual reality with a clear focus on development of algorithms for such applications. Students will learn about Fundamentals of digital audio, psychoacoustics, spatial hearing mechanism, room acoustics modelling and auralization, 3D audio reproduction, and sound synthesis methods.

Course Learning Outcomes

By successful completion of this course the students will:

• learn the basic concepts of digital audio,
• learn the basics of human auditory perception and spatial hearing,
• learn how acoustics of enclosed spaces can be modeled, simulated, and auralized and how these models

can be incorporated to computer games and virtual reality applications,
• gain knowledge on different 3D audio systems and usage scenarios,
• learn about subjective evaluation of game audio algorithms,
• have a working knowledge of different sound synthesis methods,
• gain familiarity with different audio APIs,
• develop audio algorithm development skills for games and virtual reality applications • improve their technical writing skills