Moderating Extremism
The Challenge of Combating Online Harms

size_vs_threshold_v2.jpg

The relationship between platform size and moderation

smex_num_platforms_suspended_notsuspended.png

Militant groups' handles that are banned from social media tend to be concentrated in a small number of platforms, while other platforms allow groups to maintain active accounts

Content moderation on social media platforms has become one of the most pressing and contested global policy issues. From combating misinformation and hate speech, to terrorist propaganda and other harmful material, debates over how to make online platforms safe have taken center stage. Nowhere have these efforts been more prominent than the moderation of terrorist and extremist activity on digital media platforms. “Dangerous organizations” – militant or hate-based groups, extremist organizations, and other violent movements – have become one of the main targets of moderation, experiencing mass content takedowns, account suspensions, and other sanctions. But despite the push to moderate extremism on social media, these actors continue to flourish online – advancing their cause, recruiting supporters, and inspiring violence. What explains the digital resilience of militant organizations? 

In this book, I argue that structural variation in the online information ecosystem – in particular, variation in the way social media platforms moderate content – allows militant actors to become resistant to moderation. I offer a theory of digital resilience that explains the mechanisms by which banned actors can evade moderation in a multi-platform environment characterized by inconsistent moderation policies. The theory focuses on the strategic interaction between technology companies and militant actors and explains the dynamics that allow these actors to continue advancing their goals online despite rising moderation. I present rich evidence from a variety of sources – including data on the online activity of over a hundred militant organizations, archives of banned terrorist propaganda, and a time series of platform moderation policies – to show how the digital resilience of dangerous organizations is powerfully shaped by the degree of variation in the way technology platforms police speech online.