Title: SentimentVoice
SIGGRAPH 2025 IMMERSIVE PAVILLION PREVIEW VIDEO
SentimentVoice is a VR project that uses emotion-tracking AI to amplify the narratives of immigrant communities. It can be presented as a live performance, an interactive installation, or a video documentary screening. Inspired by the pervasive use of emotional surveillance and the vulnerabilities surrounding emotion data, SentimentVoice subverts emotion AI technology, typically used for surveillance or commercial purposes, into an empathetic mediator for actively listening to the stories of immigrants. The project incorporates human voice and facial expression as the core material for the performance art, which is detected and analyzed by emotion AI.This analysis leverages various technologies, including face tracking, the ChatGPT API, voice analysis, and speech-to-text conversion.
Emotion-tracking AI, such as lie detectors and facial recognition, has often been used to create monitored environments, frequently targeting minority populations. However, it is important to recognize that emotion data can also be harnessed to cultivate empathy, foster deep connections, and promote mutual understanding.
SentimentVoice began with an oral history process, gathering authentic stories from many immigrants in the Richmond community, based on five locational memories: airports, schools, immigration offices, hospitals, and workplaces. During the live performance, one actor wears a VR headset while the other does not, establishing an intricate connection between virtual and actual space, as well as between past memories and present life. As the actors navigate the five VR locations and tell the associated stories, Emotion AI responds to their speech with visual (particles, textures, lights, etc.) and sound acknowledgement within the VR environment. The process of emotion tracking is displayed to demonstrate how our speech and facial expressions are being detected without our awareness.
Installation at hong kong city hall
Full Credits
- Creative Director: Semi Ryu
- Technical Lead, AI & VR: Miles Popiela
- AI & VR Developers: Henry Bryant, Ariana Thomas
- 3D visual producer (Modeling & Texturing): Matthew Labella
- Video Producer: Hanna Chou
- Sound Producer: Chrystine Rayburn
- Actors: Ryan Flores, Katherine Nguyen
- Script Writer: Hanna Chou, Katherine Nguyen, Ryan Flores
- Videographer: Hanna Chou, Kiara Brown, Uday Illa, Ryan Alvarado
- Interviewer: Hanna Chou, Richmond Animation Archive, Menna Hassanain
- Scientific Advisor: Alberto Cano
- 3D preliminary modeling: Menna Hassanain
- Assistant AI Engineer: Josiah Wilson
- Technical diagram: Rayna Hugo
- Interviewee: Hanna B, Kevin C, Tradie D, Kevin G, Maria G, Tina H. Dominica Ko, Sabrina Kim, Grace Kim, Magdalena Song, Sumaya S, Watt S, and more.
- Special thanks to Commonwealth Cyber Initiative, Noren Gelberg-Hagmaier, the VCU AAPIA student group, and everyone who shared their stories with us
