Sign Language Speaks Through Your Phone Camera

Google is set to revolutionize communication with its Pixel 11 lineup, integrating a new AI model called SL2T that translates sign language directly into text. This innovative feature will debut in Gboard and Live Transcribe, allowing users to communicate through ASL via their phone's camera for various tasks, including searching, texting, or interacting with Gemini.

The SL2T model analyzes 130 points on a user's face, body, and hands, processing these movements server-side to generate English text. Unlike previous methods that used intermediate 'glosses,' this direct translation captures nuances like facial expressions and spatial constructions. Google trained SL2T on over 100,000 hours of data, including ASL, and specifically accommodated one-handed and left-handed signing.

However, Google stresses that SL2T is an assistive input tool for low-stakes communication and not a replacement for interpreters. Its current limitations include potential struggles with regional signs, slang, rapid fingerspelling, and reliance on clear visual conditions. The company advises against its use in critical settings like medical or legal situations, emphasizing its role as a supplementary aid rather than a guarantee of perfect accuracy.