AI Unlocks Natural Expression in 3D Avatars

SentiPulse, in collaboration with Renmin University of China's Gaoling School of Artificial Intelligence, has launched SentiAvatar. This open-source framework aims to revolutionize 3D digital humans by enabling natural expression and interaction. Previous digital humans often suffered from stiff movements and mismatched expressions, falling into the uncanny valley.

SentiAvatar addresses this by generating human-like gestures and movements synchronized with speech. The framework leverages the SuSuInterActs dataset, containing 37 hours of multimodal conversational data for a character named SUSU. This dataset includes synchronized speech, motion, and facial expressions, specifically addressing a lack of Chinese-language data in the field.

Additionally, a proprietary Motion Foundation Model was pre-trained on over 200,000 motion sequences to learn general movement patterns. SentiAvatar's novel Plan-Then-Infill architecture separates body motion from facial expressions for enhanced realism. It achieves state-of-the-art results, generating six-second motion sequences in under 0.3 seconds for continuous, real-time interaction.

The open-source release on GitHub empowers developers worldwide to build their own 3D companions or enhance existing applications in gaming, film, and robotics, pushing the boundaries of digital human technology.