1. Google’s AI technology is being demoed to scan voice calls in real-time for financial scams, raising privacy and security concerns among experts.
2. The feature, powered by Google’s Gemini Nano AI model, is client-side scanning that could pave the way for centralized censorship once baked into mobile infrastructure.
3. European privacy experts warn that Google’s anti-scam feature could lead to social surveillance and function-creep, setting up potential risks for controlling societal behavior using AI.
Google’s demo of using generative AI technology to scan voice calls in real time for financial scams has raised concerns among privacy and security experts about the potential for centralized censorship. The feature, powered by Gemini Nano AI models, is planned to be integrated into a future version of the Android OS. This client-side scanning technology has been controversial in the past in relation to detecting illegal activities like child sexual abuse material.
Some experts, like Meredith Whittaker of Signal and Matthew Green of Johns Hopkins, warn that this technology could pave the way for widespread content scanning based on a variety of criteria, potentially leading to censorship by default. Privacy experts in Europe are particularly concerned about the implications of such technology.
Lukasz Olejnik, a privacy and security consultant, welcomes the anti-scam feature demonstrated by Google but cautions that the infrastructure could be repurposed for social surveillance, posing a threat to privacy and fundamental freedoms. He highlights the potential for AI integrated into software to control human behavior and raises questions about governance and oversight in the development of such capabilities.
Michael Veale, an expert in technology law, warns of potential function-creep resulting from Google’s conversation-scanning AI, which could lead to on-device client-side scanning for multiple purposes. The EU’s message-scanning legislative proposal has also sparked concerns about the erosion of democratic rights due to potential requirements for platforms to scan private messages by default. Critics argue that the client-side scanning technologies that may be deployed could result in numerous false positives and other vulnerabilities.
Google has yet to respond to concerns about the privacy implications of its conversation-scanning AI.