More than a dozen social media accounts have been banned by the BBC after it was exposed as having used AI to create sexually explicit content without disclosing its methods.
The twenty Instagram and TikTok accounts featured images and videos of highly sexualised black female digital avatars that were created by AI but not labelled as such, in a breach of the platforms’ guidelines.
According to the BBC, Meta, Instagram’s parent company, are currently investigating, but no actions have been taken.
The accounts were identified by the BBC and analysts from a Substack publication, Riddance, that analyses news about AI-generated media. They found that the accounts are among a growing number of Instagram and TikTok accounts that feature digitally manipulated avatars with exaggerated body parts and skin tones. They are also often dressed in revealing clothes, or swimwear, and feature racially misleading and stereotypical language.
The accounts have names containing terms such as “black” and “ebony” and comments on their posts have mentioned white men, such as “loves white men” and “why I need a white guy in my life”.
Researchers Angel Nulani and Jeremy Carrasco worked with the BBC to identify sixty accounts that used AI-generated imagery which was not labelled as such. Several carried links to paid sexually explicit content on third-party sites that disclosed the use of AI on their sites.
Nulani described the accounts as racist, saying their existence “…perpetuates a long history of the exploitation of black people,” she told the BBC.
“Their use of caricatures, race-play terminology and unrealistic depictions of black women prove they’re not concerned with our safety or wellbeing, but our ability to be capitalised as part of the online porn machine.”
Carrasco added that AI is now exploiting a “shameless” approach to “racist depictions of extremely black people”.
One content creator had her videos stolen and manipulated without her knowledge or consent. Riya Ulan said she was angry after discovering that an AI-generated video used her body and movements, overlaying her features in an extremely darkened skin tone.
“Of course my videos are all out there,” she said. “It doesn’t mean that you can just take it and steal it and post it as your own.”
The AI account that used her videos without consent has since been shut down by TikTok, though appears to still be active on Instagram.
“I’m not sure if I’m more concerned about them taking my video to promote their explicit content or [that] people actually believe in that,” Ulan said, adding that “people keep on falling for these AI models.”
According to some experts, the rise of digital and AI-powered abuse is giving way to new AI-powered forms of violence against women, including image-based abuse through deepfakes and sextortion – where non-consensual, digitally manipulated images are distributed on pornographic sites to harass women.
“What we’re seeing is a digital manifestation of larger offline truth: men target women for gendered violence and abuse,” feminist activist Laura Bates said in an interview. “But it’s also about how the tools facilitate that abuse.”