1 Chinese-Speaking 3D Talking Head Project No: H08040 Sang Siew Hoon Supervisor: Dr Ng Teck Khim 2 System Objectives create a realistic Chinese-speaking 3D talking model create a realistic Chinese-speaking 3D talking model -automatic -accurate and realistic -portable 3 Problems Lack of references on Chinese Lack of references on Chinese Involves various fields Involves various fields -Face modeling -Text-to-speech -Face Animation (Focus) 4 System Overview 5 modules 1. Face Model Preparation 2. Chinese Visemes 3. Text-to-Speech (TTS) 4. Animation 5. Rendering 5 6 Face Model Preparation 7 Face Modeling 3DS Max Face Modeling 3DS Max 8 Face Model Preparation Defining MPEG-4 feature points Defining MPEG-4 feature points 9 Face Model Preparation Assigning lip vertices Assigning lip vertices 10 Face Model Preparation Cloning morph targets Cloning morph targets FAP#3 open_jawFAP#4 lower_t_midlip 11 Chinese Visemes Visemes - the visual equivalent of a phoneme 1. Classification of Chinese Syllables 2. Definition of Chinese Visemes 3. Refining Dynamic Visemes 12 Chinese Visemes Definition of Chinese Visemes Definition of Chinese Visemes 13 Chinese Visemes Definition of Chinese Visemes Definition of Chinese Visemes 14 Chinese Visemes Definition of Chinese Visemes Definition of Chinese Visemes 15 Chinese Visemes Refining Dynamic Visemes Refining Dynamic Visemes 1. + e only replace er 2. zh, z, y + I drop i 3. d, l, g, j, zh, z, y, w + u | drop 3. d, l, g, j, zh, z, y, w + u | drop 4. j, y + ending with an replace the an by en 5. j + complex finals headed by i and followed by more than one drop i in complex finals 16 Text-to-Speech 17 Text-to-Speech 18 Animation 1. Coarticulation 2. Automatic Generation of Face Animation 3. Enhancing Realism 19 Animation Enhancing realism Enhancing realism - Eye blinks - Eyebrow raising - Gaze - Head Rotation 20 Results Accurate and realistic Accurate and realistic No discrepancies for new face models No discrepancies for new face models Automated results need slight manual intervention Automated results need slight manual interventionhttp://www.slamstudio.com/HYP/videos 21 Conclusion Contributions Contributions 1. own system to define Chinese visemes - adds to the current research works of Chinese talking heads (currently limited). Chinese talking heads (currently limited). 2. an automatic Chinese lip-synchronization system - saves an animator much time and effort 22 Conclusion System Limitations System Limitations 1. TTS limitation 2. FAPs conflicts 3. Not integrated inconvenient for untrained users 23 Conclusion Future Works Future Works 1. Automatic audio signal processing 2. Resolve FAPs values conflicts 3. Automatic feature points assignment system 4. Integration of system