As usual, unbridled “free speech”, voluntary blindness, minimization of harm, and inexistent enforcement of laws against gender based violence invariably impact women and girls. Given that there has been little political or judicial will to stop intimate violence, it is hardly surprising to see generative AI being hijacked to produce ever more nonconsensual intimate images of women and girls, as is the case with the latest anti-social trend of “undress technology” being widely used in schools by teenage boys who undress their teachers and classmates for the purpose of causing long-lasting harm and inciting girls to commit suicide. While videos are harder to produce, the creation of images using “undress” or “nudify” websites and apps has become commonplace.
Big tech and investors are complicit and should be subjected to criminal investigations, but aren’t. An alarming report by 404 Media shows that violence through deepfake technology is intentionally promoted and knowingly encouraged by Big Tech platforms be it via targeted ads on social media or directly in app stores as it appears on the top of searches.
As if this weren’t enough, WIRED reports that Big Tech platforms further facilitate violence against women by allowing people to use their existing accounts to join the deepfake websites. For example, Google’s login system appeared on 16 such websites, Discord’s appeared on 13, and Apple’s on six. X’s button was on three websites, with Patreon and messaging service Line’s both appearing on the same two websites. The login systems have been used despite the tech companies terms and conditions that state developers cannot use their services in ways that would enable harm, harassment, or invade people’s privacy.
Sign-in APIs are tools of convenience. We should never be making sexual violence an act of convenience. We should be putting up walls around the access to these apps, and instead we’re giving people a drawbridge.”
“This is a continuation of a trend that normalizes sexual violence against women and girls by Big Tech,” says Adam Dodge, a lawyer and founder of EndTAB (Ending Technology-Enabled Abuse).
After being contacted by WIRED, spokespeople for Discord and Apple said they have removed the developer accounts connected to their websites. Google said it will take action against developers when it finds its terms have been violated. Patreon said it prohibits accounts that allow explicit imagery to be created, and Line confirmed it is investigating but said it could not comment on specific websites. X did not reply to a request for comment about the way its systems are being used.
The tech company logins are often presented when someone tries to sign up to the site or clicks on buttons to try generating images. It is unclear how many people will have used the login methods, and most websites also allow people to create accounts with just their email address. However, of the websites reviewed, the majority had implemented the sign-in APIs of more than one technology company, with Sign-In With Google being the most widely used. When this option is clicked, prompts from the Google system say the website will get people’s name, email addresses, language preferences, and profile picture.
“In order to use Sign in with Google, developers must agree to our Terms of Service, which prohibits the promotion of sexually explicit content as well as behavior or content that defames or harasses others,” says a Google spokesperson, adding that “appropriate action” will be taken if these terms are broken. Other tech companies that had sign-in systems being used said they have banned accounts after being contacted by WIRED.
“We must be clear that this is not innovation, this is sexual abuse. These websites are engaged in horrific exploitation of women and girls around the globe. These images are used to bully, humiliate, and threaten women and girls”, says David Chiu, San Francisco’s city attorney.
This fiasco has prompted San Francisco’s city attorney to file a lawsuit against undress and nudify websites and their creators. Chiu says the 16 websites his office’s lawsuit focuses on have had around 200 million visits in the first six months of this year alone. The lawsuit brought on behalf of the people of California alleges that the services broke numerous state laws against fraudulent business practices, nonconsensual pornography and the sexual abuse of children. But it can be hard to determine who runs the apps, which are unavailable in phone app stores but still easily found on the internet.
The undress websites operate as shadow for profit businesses and are mainly promoted through criminal platforms like Telegram who notoriously push child porn and human trafficking worldwide under the guise of “free speech”. The websites are under constant development: They frequently post about new features they are producing—with one claiming their AI can customize how women’s bodies look and allow “uploads from Instagram.” The websites generally charge people to generate images and can run affiliate schemes to encourage people to share them; some have pooled together into a collective to create their own cryptocurrency that could be used to pay for images.
As well as the login systems, several of the websites displayed the logos of Mastercard or Visa, implying that banks are entirely on board with deepfake technology although they claim otherwise. Visa did not respond to WIRED’s request for comment, while a Mastercard spokesperson says “purchases of nonconsensual deepfake content are not allowed on our network,” and that it takes action when it detects or is made aware of any instances.
On multiple occasions, the only time tech companies and payment providers intervene is when pressured by media reports and requests by journalists. If there is no pressure, it is business as usual in the realm of violence against women and girls. And we all know it is a lucrative one.
“What is concerning is that these are the most basic of security steps and moderation that are missing or not being enforced. It is wholly inadequate for companies to react when journalists or campaigners highlight how their rules are being easily dodged. It is evident that they simply do not care, despite their rhetoric. Otherwise they would have taken these most simple steps to reduce access.” Clare McGlynn, law prof at Durham University
No, they don’t care. We must ban speech altogether and start from scratch.