Both of the world’s dominant app stores are under growing pressure this year over what they’re allowing AI-powered apps to do — and, in a notable escalation this week, a city government is now formally demanding they act. San Francisco’s city attorney sent cease-and-desist letters to both Apple and Google, ordering them to remove a specific category of harmful AI apps from their platforms within 28 days or face potential civil penalties.
That legal action is the sharpest point in a broader pattern that’s been building for more than a year: both companies tightening developer guidelines, pulling flagged apps, and facing mounting criticism that their enforcement has been reactive rather than proactive. Here’s what’s actually driving the pressure, and how each company has responded so far.
The City Attorney’s Legal Action
San Francisco City Attorney David Chiu’s office sent formal cease-and-desist letters to both Apple and Google this week, demanding the removal of a specific set of apps that use AI to generate non-consensual sexualized images of real people from photos. The letters cover 13 apps under investigation — eight distributed through Apple‘s App Store and five through Google Play — and note that California law prohibits supporting services that create this kind of deepfake content.
Chiu’s office says both companies have “been on notice” about their role in processing payments connected to this activity for close to a year, but argues neither has acted with sufficient urgency. The letters state plainly that both companies have likely earned “millions of dollars in fees” from apps offering these services, and give both companies 28 days to respond or risk facing civil penalties. This isn’t the city attorney’s first action in this space either — the same office has previously taken legal action against 16 separate deepfake websites operating outside the app store ecosystem entirely.
What an Independent Investigation Found
The city’s action follows an earlier report from the Tech Transparency Project (TTP), which found that both companies’ search and advertising systems were, in some cases, actively directing users toward this same category of app. According to data cited from mobile analytics firms, the apps identified in that report had collectively been downloaded more than 483 million times and generated over $122 million in revenue.
A particularly serious finding involved age ratings: the report identified 31 of these apps that were rated as suitable for all ages, including minors. Following the report’s publication, Apple removed 15 of the flagged apps and Google suspended several others, with both companies describing their enforcement efforts as ongoing. Both companies maintain formal developer policies prohibiting this kind of content — Google has reiterated publicly that it does not allow apps containing sexual content and takes action whenever violations are identified.
Google’s Developer Guidance Update
Separately, Google has issued updated developer guidance specifically for AI-powered apps distributed through Google Play, aimed at reducing this same category of harmful content across the platform. Under the new guidance, apps offering AI features are required to actively prevent the generation of restricted content — including sexual content and violence — and must provide users a way to flag offensive content they encounter.
The guidance also requires developers to “rigorously test” their AI tools and models to ensure they respect user safety and privacy before distribution. Notably, Google’s policy update extends to marketing itself: if an app’s advertising or promotional materials suggest it’s capable of generating this kind of harmful content, the app can be removed from Google Play regardless of whether it’s technically able to do so — closing a loophole where apps might advertise capabilities they don’t fully deliver, purely to attract downloads.
Apple’s Separate Privacy Crackdown
Apple has also been tightening its App Review Guidelines around AI more broadly, though on a different front. In an update to its developer guidelines, Apple added a requirement that apps must disclose and obtain user permission before sharing personal data with third-party AI systems — a change that arrived ahead of Apple’s own plans to introduce an AI-upgraded version of Siri, reportedly powered in part by Google’s Gemini technology.
That update reflects a broader concern distinct from the deepfake and nudify app problem: ensuring that apps aren’t quietly funneling user data to outside AI companies without clear consent, even as Apple simultaneously builds its own AI partnerships into core platform features.
Why Enforcement Keeps Falling Short
Despite both companies maintaining policies that explicitly prohibit this content, researchers and regulators have repeatedly found the same category of app resurfacing. Schools across the US have reported growing problems with AI-generated harmful images being circulated among students for bullying and harassment — a problem serious enough that one incident, involving an AI-generated deepfake of a school principal, led to an arrest. Reports indicate the problem has reached even middle schools in some cases.
Part of the challenge appears structural: developers can rebrand, resubmit, or slightly alter a previously-removed app to get back onto a platform, while both companies’ automated review and advertising systems have, at times, actively surfaced this content to users searching for it — a failure mode a platform explicitly can influence in a way it can’t fully control determined individual bad actors.
The Bigger Pattern Behind This
This isn’t the first time Apple and Google have faced pressure over how they police their app stores, though the AI dimension has added new urgency. Google has previously disclosed blocking millions of apps from Play in a single year for policy violations, and both companies maintain large, ongoing content moderation operations. What’s different now is the specific combination of generative AI capability with a payment and distribution system used by billions of people — meaning a harmful AI tool doesn’t need its own website or infrastructure to reach a mass audience; it just needs to get past app store review once.
That dynamic is likely why regulators like San Francisco’s city attorney are now applying direct legal pressure rather than relying solely on the platforms’ own enforcement processes — treating Apple and Google less as neutral distributors and more as parties with genuine responsibility for what their stores allow through.
Conclusion
Apple and Google’s tightening rules around AI apps reflect a genuine, if reactive, pattern: both companies have real, stated policies against harmful AI-generated content, but both have also repeatedly been shown — by independent researchers and now by a city government — to be enforcing those policies inconsistently. With a 28-day deadline now on the clock from San Francisco’s city attorney, the next few weeks should offer a clearer signal of whether this results in meaningfully tighter enforcement, or simply another round of app removals followed by new apps taking their place.
