An Interactive Mixed Reality Framework for Virtual Humans

Egges, A., Papagiannakis, G., Magnenat-Thalmann, N., “An Interactive Mixed Reality Framework for Virtual Humans”, Cyberworlds 2006, IEEE Computer Society, Lausanne, September 2006



In this paper, we present a simple and robust Mixed Re- ality (MR) framework that allows for real-time interaction with Virtual Humans in real and virtual environments un- der consistent illumination. We will look at three crucial parts of this system: interaction, animation and global illu- mination of virtual humans for an integrated and enhanced presence. The interaction system comprises of a dialogue module, which is interfaced with a speech recognition and synthesis system. Next to speech output, the dialogue system generates face and body motions, which are in turn man- aged by the virtual human animation layer. Our fast an- imation engine can handle various types of motions, such as normal key-frame animations, or motions that are gen- erated on-the-fly by adapting previously recorded clips. All these different motions are generated and blended on-line, resulting in a flexible and realistic animation. Our robust rendering method operates in accordance with the previ- ous animation layer, based on an extended for virtual hu- mans Precomputed Radiance Transfer (PRT) illumination model, resulting in a realistic display of such interactive virtual characters in mixed reality environments. Finally, we present a scenario that illustrates the interplay and application of our methods, glued under a unique framework for presence and interaction in MR. [PDF]