The Politics of Social Media Manipulation (deel 2) (bijlage bij 30821,nr.91) - Hoofdinhoud

Officiële titel	The Politics of Social Media Manipulation (deel 2) (bijlage bij 30821,nr.91)
Document­datum	18-10-2019
Publicatie­datum	22-10-2019
Nummer	2019D41922
Kenmerk	30821, nr. 91
Externe link	origineel bericht
Originele document in PDF

sources like NOS, GeenStijl, and VPRO Zondag met Lubach. On 4chan/pol/, however, they are far more extreme and potentially harmful. The most-posted channel is SouthFront, dedicated to videos on the Syrian civil war. Below that is Stefan Molyneux, a popular Canadian YouTuber who promotes “scientific racism” and white supremacist views. Further down the list are

(hyper)partisan news channels like Fox News as well as the Russian RT and Ruptly. Other farright YouTubers and channels also appear, like Paul Joseph Watson and Rebel Media, as well as some left-leaning channels like The Young Turks and VICE. Together, the channels referred to by Dutch posters are thus of a hyperpartisan, sometimes far-right makeup. As such, YouTube videos on Dutch subreddits seem to align with consumption of “established” and “traditional” news media outlets, while those on 4chan/pol/ show a highly hyperpartisan and polarised landscape.

Conclusions

Despite the frequent characterisation of Reddit and 4chan as “alternative” zones on the Web, the results presented in this text generally do not reveal a large share of alternative news networks spreading disinformation within the platforms, at least in a Dutch context. Despite a few instances of pro-Russian websites like Novini and one suspicious Reddit account, coordinated campaigns of malicious users posting links to disinformation seem largely absent. Dubious content can certainly be discerned but compared to overall activity (as shown in section two) it should be considered fairly marginal within the spaces we scrutinised. Reddit seems especially resilient against the circulation of junk news. In turn, the characterisation as actors within 4chan and parts of Reddit as influential “agenda setters” should therefore likely be taken with a grain of salt.

Figure 20. The top 1008 most-posted YouTube videos in Dutch subreddits. Black labels denote deleted videos/channels. Ranked left to right, top to bottom. Data source: 4CAT, Pushshift, and YouTube API.

 141 Figure 21. The top 1008 most-posted YouTube videos in Dutch subreddits, with video categories as an overlay.

Black labels denote deleted videos/channels. Ranked left to right, top to bottom. Data source: 4CAT, Pushshift, and YouTube API. Image wall.

 142 Figure 22. The top 1008 most-posted YouTube videos in 4chan/pol/in posts with a Dutch country flag. Black labels denote deleted videos/channels. Ranked left to right, top to bottom. Data source: 4CAT and YouTube

API. Image wall.

Figure 23. The top 1008 most-posted YouTube videos in 4chan/pol/in posts with a Dutch country flag, with video categories as an overlay. Ranked left to right, top to bottom. Black labels denote deleted videos/channels.

Data source: 4CAT and YouTube API. Image wall.

 143 4chan/pol/ - top 25 most-occurring channels Reddit - top 25 most-occurring channels

channel count channel count channel count channel count

FOX 10 South Front 191 Phoenix 49 AFC Ajax 476 AT5 38

Stefan Molyneux 177 sanderson1611 46 VitesseTV 269 FvD Meems 37

Forum Fox News 156 PewDiePie 41 Democratie 143 De Speld 36

Paul Joseph Omroep RT 155 Watson 39 PowNed 107 Football-Oranje 32

The White Acts17Apologe Politie House 136 tics 38 #PRO247 95 LISSAUER 31

BRAXATOR Ruptly 125 Rebel Media 37 ES 82 vpro.nl 30

Right Side Broadcasting VPRO Zondag

Network 112 VICE 37 met Lubach 69 De Telegraaf 30

Omroep PowNed 108 Fullwhiskey 36 GeenStijl 67 NOS op 3 30

U.S. Department of Cafe

State 76 VICE News 34 Weltschmerz 51 Hoop Stront 29

CNN 68 ABC News 34 NOS 51 PVVpers 28

Forum Democratie 67 DeroVolk 33 WNL 48 RTL Z 27

Politie Den Fox Business 65 corbettreport 31 Xbox 47 Haag 24

The Young Turks 56 TopNotch 43

Table 3. The most-occuring YouTube channels from all YouTube links posted in the Dutch Reddit and

4chan/pol/ samples. Data source: 4CAT, Puhshift, and YouTube API. Timeframe: from 01-Dec-2015 to 01- Jun-2016.

What can be observed, however, are the types of junk news that can be characterised as hyperpartisan, especially on 4chan/pol/. This appeared mostly through links to popular tendentious and hyperpartisan blogs like The Post Online and De Dagelijkse Standaard, but also the more clearly “fake” (in the sense of conspiratorial) NineForNews. This right-wing bias is expected for 4chan/pol/ due to its infamy as a far-right hub; for Reddit it is more notable because we took a politically diverse range of URLs and subreddits as a starting point. While labelling these websites as “fake” is problematic, they do indicate a non-negligible presence of polarising content. Indeed, section four showed that the most-engaged-with articles from these websites often concern topics like migration and Islam, instead of other geopolitical events like Russian interference.

 144 Nonetheless, mainstream sources such as NOS.nl remain popular linked-to domains on both

Reddit and 4chan/pol/. This is somewhat counterintuitive since it has been argued the “fringe” characterisation of these pseudonymous and anonymous spaces implies their users find knowledge in different epistemological drawers. Despite these assumptions, the prevalence of mainstream sources shows they have at least some authority within these online spaces. It is important to note, however, that we have not considered how these mainstream websites are discussed. Likely, domains like NOS.nl are considered on Reddit as a trustworthy source, while on 4chan/pol/ it might be referenced purely to ridicule it or to portray it as “fake news” itself - as is discussed elsewhere in this volume.

One should furthermore not be blinded by exclusively considering websites devoted to reporting on current affairs as the sole source of news, as YouTube emerged as a particularly big “new” player in relation to news consumption and circulation, especially on 4chan/pol/. On Reddit,

“News & Politics” videos on YouTube are estimated to form the second-largest source, while on 4chan/pol/, they are estimated to strongly outperform any other news source. From a brief exploration of the YouTube channels posted on both platforms, it seems Dutch Reddit is largely linking to fairly established sources, like PowNed, Zondag met Lubach, and NOS, while on

4chan/pol/, alternative, hyperpartisan, and problematic information channels emerged, like Mike Cernovich and RT. As such, non-Dutch YouTube content might have a “radicalising” role on

Dutch users within certain Internet forums.

Since this report concerns the news sources linked to by actors on Reddit and 4chan, it does not shed light on the grassroots production of alternative news or conspiracies within these spaces. As

Tuters et al. (2018) show in relation to the Pizzagate conspiracy, the wildest theories can be cooked-up in these spaces through a short burst of a “butterfly effect” of “bullshit”, unobservable when merely considering the prominence of URLs. A more holistic approach, also taking into account text and images, could thus aid in further contextualising the current “fake news” debate. Instead of identifying isolated issues of “fake news”, such broader approaches could tackle the interwoven problematics surrounding the circulation of “junk news” (Venturini 2019) and “network propaganda” (Benkler et al. 2018), from the conspiracist mindset of

“virality-oriented subcultures” to the prevalence of polarising hyperpartisan content.

 145 Appendices

Appendix I: Compiled list of Dutch Subreddits

ADODenHaag,AjaxAmsterdam,Alkmaar,Aluhoedjes,Amersfoort,Amster dam,AmsterdamEnts,Appiememes,Arnhem,Aruba,Assen,avd,AZAlkmaar ,Bassie_en_Adriaan,BeermoneyNL,BeNeLux,Bier,Binnenhof,Bitcoin NL,Boeken,Bonaire,BuurmanEnBuurman,CariceVanHouten,CelebsNL,C irkeltrek,CreatieveKoppen,Curacao,de_thierry,de_thierry,DeCor respondent,DeGraafschap,Delain,Delft,Depressie,DeSpeldOfNietD eSpeld,DeStaat,DeStagiair,DeTandenborstel,DirkJan,Dordrecht,D oucheGedachten,DoutzenKroes,Drenthe,Duindorp,Dumoulin,Dunglis h,dutch,Dutch,DutchBoardgames,DutchComedy,DutchDesign,DutchEn ts,DutchFIRE,DutchHipHop,DutchHouse,DutchKeto,DutchMusic,Dutc hPoetry,DutchProblems,DutchSkincare,DutchTech,Eindhoven,Elfst edentocht,Enschede,Epica,Eredivisie,Ethtradernl,FCGroningen,F CTwente,FCUtrecht,Feyenoord,Formule1,Forum_Democratie,FreeDut ch,Frisia,Frysk,GekkeJongens,Geldzaken,Geschiedenis,Gezellig, Glitterplaatjes,groenlinks,Groningen,Haarlem,HanzeMemes,Heilz ameMeems,Hulpdiensten,ik_ihe,JuridischAdvies,Kamerstukken,Kat holiekeNederlanden,Kibbeling,KNVB,Koffie,Kut_Doen_Op_Tinder,K utleven,Kutreclames,LearnDutch,LeCutInsideMan,Leiden,Leraren, LimburgMan,Lowlands,Maastricht,MamaAppelsap,Marktplaats,MaxV, Medejongeren,meerderheidnederland,Metal_NL,Motorfietsen,NACBr eda,NEC,Nedercringe,Nederporn,NepParlement,netherlands,Nether landsPics,NietDeSpeld,Nijmegen,NLvsFI,Nuenen,NuJijInActie,oek raineukraine,ossem,otonde,Papgrappen,ParadoxPlaats,PECZwolle, PodcastNed,PokemonGoNL,Poldersocialisme,Politiek,Politiekmeme s,PSV,RijmenDichten,RMTK,RodaJC,RomeeStrijd,Rotterdam,Saba,SC Cambuur,SCHeerenveen,ScoutsNL,SportNL,Spyker,StefanieJoosten, Strips,Stroopwafels,StudyInTheNetherlands,SXM,SylvieMeis,Tene nkrommend,The_Klaver,The_Wilders,TheHague,theNetherlands,theN etherlandsFree,theNetherlandsNature,Tiesto,tokkiefeesboek,tok kiefeesboek,TokkieFeesboek,Top2000,TUDelft,TuurlijkIsDatEenDi ng,Twente,Utrecht,VeganNL,Veluwe,VitesseArnhem,Voetbalnieuws, VraagDerNederlanden,VraagHetAanTonyQuark,Wetenschap,WIDM,With inTemptation,XboxNederland,Zitkamer,ZonderContext,Zwolle

Table 4. Compiled list of Dutch subreddits.

 146 Appendix II: Expert List of Dutch Junk News Domains

name domain_name category

Opiniez opiniez.com hyperpartisan

Stop de Bankiers, stopdebankiers.com hyperpartisan

t Pallieterke pallieterke.net hyperpartisan

E.J. Bron ejbron.wordpress.com hyperpartisan

Dagelijkse Standaard, dagelijksestandaard.nl hyperpartisan

Climategate climategate.nl hyperpartisan

De Staat van het klimaat destaatvanhet-klimaat.nl hyperpartisan

JDreport.com jdreport.com hyperpartisan

tpook.nl tpook.nl clickbait

Nine for news ninefornews.nl conspiracy

Daily Paper dailypaper.org hyperpartisan

Parra parra.nu clickbait

Viraaltjes viraaltjes.nl clickbait

about media aboutmedia.nl clickbait

Martin Vrijland martinvrijland.nl conspiracy

The Loyalist loyalist.nl conspiracy

desportgek desportgek.nl clickbait

Even Delen evendelen.net clickbait

nietbarkie.nl nietbarkie.nl clickbait

hardewaarheid.nl hardewaarheid.nl clickbait

tendentious The Post Online tpo.nl hyperpartisan

Saltmines.nl saltmines.nl hyperpartisan

eunmask.wordpress.com eunmask.wordpress.com hyperpartisan

novini.nl novini.nl hyperpartisan

niburu.nl niburu.nl conspiracy

React nieuws reactnieuws.net hyperpartisan

DMLplus dlmplus.nl conspiracy

martinvrijland.nl martinvrijland.nl conspiracy

world unity worldunity.me conspiracy

cultuur onder vuur cultuurondervuur.nu hyperpartisan

volks nieuws uit Amsterdam noir volksnieuwsuitamsterdamnoir.com conspiracy

stop pas familie drama stoppasfamiliedrama.blogspot.com conspiracy

Obed Brinkman obedbrinkman.noblogs.org hyperpartisan

veren of lood verenoflood.nu hyperpartisan

De fouten van Rutte defoutenvanvvdrutte.nl hyperpartisan

Finding voices finding-voices.blogspot.com conspiracy

 147 ik was haren ikwasinharen.nl hyperpartisan

Piet Kei pietkei.nl conspiracy

bewiseman bewiseman.nl hyperpartisan

Alternatieve Media Nederland alternatievemedianederland.com hyperpartisan

Apokalypsnu apokalypsnu.nl conspiracy

Don Quijotte donquijotte.wordpress.com conspiracy

Drimble drimble.nl hyperpartisan

Fenixx fenixx.org hyperpartisan

Hector Reban hectorreban.wordpress.com hyperpartisan

Herstelde Republiek herstelderepubliek.wordpress.com hyperpartisan

Kremlin Troll kremlintroll.nl hyperpartisan

Magilando magilando.wordpress.com conspiracy

Niburu niburu.co conspiracy

Absolute Duality nl.absoluteduality.com conspiracy

Stan van Houcke stanvanhoucke.blogspot.com hyperpartisan

Stelling stelling.nl conspiracy

Tref tref.eu hyperpartisan

Want to know wanttoknow.nl conspiracy

Xandernieuws xandernieuws.punt.nl hyperpartisan

APost apost.com clickbait

Best Gezond bestgezond.nl clickbait

Bewust Nieuws bewustnieuws.nl conspiracy

Blik Op NOSjournaal blikopnosjournaal.blogspot.nl hyperpartisan

Bovendien bovendien.com conspiracy

Brekend Nieuws brekendnieuws.nl conspiracy

Dagelijks.nu dagelijks.nu clickbait

Dagelijkse Krant dagelijksekrant.nl clickbait

De Stille Waarheid destillewaarheid.nl hyperpartisan

Earth Matters earth-matters.nl conspiracy

Ella’ster ellaster.nl conspiracy

Health Bytes healthbytes.me conspiracy

Healthwatch

gezondheidswaakhond healthwatch.nu conspiracy

Leeshetnu leeshetnu.nl clickbait

Lekkerwonen lekkerwonen.org clickbait

LikeMag likemag.com clickbait

Lijstverse lijstverse.nl clickbait

Live kijken livekijken.nl clickbait

Nieuwsdump nieuws-dump.nl clickbait

 148 Not100 not100.nl clickbait

Ongelooflijke Verhalen smullen-maar.nl clickbait

Prankster prankster.nl clickbait

Revolutionair Online revolutionaironline.com hyperpartisan

Snuggerd snuggerd.nl clickbait

Time 2 Wake Up time2wakeup.me hyperpartisan

Tips & Weetjes tipsenweetjes.nl clickbait

Tis Wat tis-wat.nl clickbait

TrendBuzz trendbuzz.nl clickbait

Trendnieuws trendnieuws.nl clickbait

Trendnova trendnova.nl clickbait

United-Lightworkers united-lightworkers.be conspiracy

Vaccinatieraad vaccinatieraad.nl conspiracy

Viraalpunt viraalpunt.nl clickbait

Viral Mundo viralmundo.nl clickbait

Viraaltje Viraaltje.nl clickbait

Vrouwen Dingen vrouwendingen.com clickbait

Vrijspreker vrijspreker.nl hyperpartisan

tendentious The Post Online - Politiek politiek.tpo.nl hyperpartisan

Erkenbrand erkenbrand.eu hyperpartisan

Das Kapital daskapital.nl hyperpartisan

Glop glop.nl hyperpartisan

Table 5. Junk news categorisation (expert list). Edited and enhanced list originating from Hoax-Wijzer.

 149 Appendix III: Metrics on domains shared on Reddit and 4chan/pol/

Reddit 01-12-2015 to 01-06-

2019 OPs Category Count Percentage

Positive Negative Percentage Mainstream 5255 89.9%

News 5959 27594 21.64% Other 580 9.7%

Dutch news 5557 402 93.3% Hyperpartisan 24 0.4%

Dutch junk news 24 5935 0.4% Disinformation 0 0%

 Clickbait 0 0%

 Conspiracy 0 0% Table 6 . Metrics for the proportions of news, Dutch news, Dutch junk news, and categories in posts on Dutch language Subreddits, 01-12-2015 to 01-06-2016.

Reddit 01-12-2015 to 01-06-

2019 OPs Category Count Percentage

Positive Negative Percentage Mainstream 10399 71.5%

News 14541 87301 16.6% Other 1414 9.5%

Dutch news 3403 11138 23.41% Hyperpartisan 2091 14.4%

Dutch junk news 2809 11732 19.3% Disinformation 241 1.7%

 Clickbait 45 0.3%

 Conspiracy 351 2.4%

Table 7. Metrics for the proportions of news, Dutch news, Dutch junk news, and categories in posts on

4chan/pol/ with a country flag from the Netherlands, 01-12-2015 to 01-06-2016.

 150 Appendix IV: Most-posted URLs from posts containing links to RT.com and Sputnik on

4chan/pol/

Title URL Amount of appearances

https://www.rt.com/news/32924 9

Stabbing death of 15yo schoolboy by 3-sweden-migrant-stabbed‘Arab

migrant’ classmate in Sweden teenager/

sparks outrage

Sweden: Rape Capital of the West https://www.gatestoneinstitute.or 9 g/5195/sweden-rape

https://www.rt.com/news/36941 9 5-sweden-refugees-rape-afgan

Sweden charges 5 teenage refugees with boy/

beating, gang-raping boy for over an hour

https://www.rt.com/news/33727 8 6-belgium-nuclear-guard-killed/

Belgian prosecutor’s office denies terrorist track in murder of guard at nuclear center

Sex Slave Found Chained in Basement http://speisa.com/modules/article 8 of Immigrant Cafe in Sweden s/index.php/item.3584/sex-slavefound-chained-in-basement-of- immigrant-cafe-in-sweden.html (now offline)

Table 8. Most occuring URLs from posts containing links to RT.com and Sputnik by posts with a Dutch country flag on 4chan/pol/. Derived with 4CAT.

 151

FAKE NEWS AND THE DUTCH YOUTUBE

POLITICAL DEBATE SPACE

Marc Tuters

Fake News and the Dutch YouTube Political Debate Space

Marc Tuters

Introduction: YouTube as radicalising platform

On 1 February 2019, De Volkskrant and De Correspondent published a much anticipated report on YouTube as a radicalization platform: “Leidt het algoritme van YouTube je naar extreme content?” (Translated: Does the YouTube algorithm lead you to extreme content?) (Bahare et. al. 2019). Drawing on data analysis produced by some of the same authors of this current report, it sought to investigate the extent to which YouTube functioned as an engine for online

“radicalization” (Tufekci 2018; Holt 2017). As these and other reports claimed, YouTube appears to be playing a significant role in the development of a new antagonistic culture of debate, in which an “alternative influence network” is said to have the capacity to shape public opinion, especially amongst a demographic of young and politically rightward leaning men

(Lewis 2018). Amongst the figures who have risen to prominence through this YouTube debate culture, is for example the now internationally well-known, Canadian academic psychologist

Jordan Peterson. Peterson is often viewed as a conservative political figure, even as a member of the so-called “alt-right” (Lynskey 2018). This latter term, which stands for “alternative right”, gained popularity in the aftermath of the 2016 US election as a means of describing a seemingly new breed of conservative online activism that brought together a diverse array of actors united against the perceived hegemony of “politically correct” liberal values, often through a jokey and transgressive style (Hawley 2017; Heikkilä 2017; Nagle 2017). Whilst Peterson has refuted an association with the alt-right, in consulting how the YouTube algorithm itself categorizes

Peterson it would appear that the platforms nevertheless still views him in this light. How exactly this categorization works is inscrutable to all but the owners of the platform. And while it should not be taken as definitive proof of what a given channel is about, we can nevertheless assume that YouTube’s categorization does reflect some essential aspect of its bottom line, which is to keep the most people watching for the longest time possible.

The present research report uses the same platform-centric categorization method as introduced above, applying it to studying the space of Dutch parliamentary political debate on YouTube.

While initially motivated by the question of how this space engages with the issue of “fake news”, the report however moves away from defining fake news as disinformation (which is to say the deliberate manipulation of facts) towards conceiving of it in terms of a form of

“hyperpartisan” information as produced by “openly ideological web operations” (Herrman

2016). This latter conception of fake news is furthermore also resonant with the redefinition of the term as it has begun to be appropriated by politicians around the world in order to describe news organizations whose coverage they find “disagreeable” (Wardle and Derakshan 2017, 16)

— notably by Donald Trump who often refers to “establishment” media outlets such as CNN and the New York Times as fake news (Weisman 2018). In the European context, where laws such as the German Netz DG have been passed at the national level rendering platforms responsible for policing this problem, such critics have framed the attempt at regulating fake news as a

“blueprint” for state censorship (Wardle and Derakshan, 2017: 71). In spite of these controversies, the bewildering issue of fake news, entangled as it is together with broader changes in political and media spheres at a variety of levels, remains relatively understudied

 152 outside of the American context — the latter which is in many ways quite unique for a variety of factors (Benkler et al., 2018: 381-387).

Whilst the precise mechanisms of YouTube’s algorithms are unknown, what is clear is that they are designed to optimize “engagement,” defined in terms of “views” as well as the number of

“comments”, “likes”, and so forth (Covington et al., 2016). In recent years, YouTube’s algorithm has been critiqued as creating a so-called “rabbit-hole effect” (Holt, 2017), whereby the platform’s algorithms, as mentioned above, have been accused of recommending ever more extreme content, in an effort to keep viewers engaged. It has thus been argued that this particular environment has helped to draw audience from the mainstream towards the fringe. Along these lines, it has indeed been argued that, on YouTube, “far-right ideologies such as ethnonationalism and anti-globalism seem to be spreading into subcultural spaces in which they were previously absent” (Marwick and Lewis, 2017: 45). Academic researchers exploring this phenomenon have, for instance, found that YouTube’s “recommendation algorithm” has a history of suggesting videos promoting bizarre conspiracy theories to channels with little or no political content

(Kaiser and Rauchfleisch, 2018). Beyond this current “radicalization” thesis, for some years new media scholars have observed that YouTube appears to multiply extreme perspectives rather than facilitating an exchange or dialogue between them — as for instance observed in an earlier audience reception study of polemical documentary produced by the Dutch parliamentarian

Geert Wilders and published to YouTube (van Zoonen et al., 2011).

We may perhaps want to consider the growth of a new combative and conspiratorial culture of debate on YouTube, as documented by these more recent YouTube studies, in the context of broader global political shifts that have been picking up pace in the latter part of the 2010s, the latter which may be referred to under the umbrella term of “national populism” (Eatwell and

Goodwin, 2018). Referred to as “thin ideology” (Mudde and Kaltwasser 2017), populism is characterized by a suspicion of the “elite” as well as a purist notion of the “general will” of the true people, the latter which is not necessarily equivalent to the democratic electorate (Muller,

2016). Recent new media scholarship has convincingly demonstrated how such populist anti-elite sentiment translated readily into an embrace of alternative news media, particularly in the US context in which the rise of an “alternative partisan news system” is said to have played a crucial role in the last presidential election (Benkler et al., 2018). While there exists right and left variants of the concept, right-wing populists tend to have an advantage in speaking to nationalist issues

(Goodwin and Eatwell, 2018). In the analysis of political scientists Matthew Goodwin and Roger Eatwell, national populism can be characterized by four factors, that they call the “four D’s”.

These are a distrust in the liberal “establishment”, the destruction of long-held communal identity owing to forces of globalization, the relative deprivation as “neoliberal” economics leads to a rise in inequality and finally the political de-alignment from traditional political parties. Whatever the political valence of national populism going forward, Goodwin and Eatwell conclude that these four factors are destined to have “a powerful effect on the politics of many Western countries for many years to come” (Goodwin and Eatwell, 2018).

Fakeness and hyperpartisanship

Thus far the problem of fake news has primarily been studied in the context of Anglo-American national populism, specifically the political communication surrounding the Brexit referendum

 153 and the insurgent Trump campaign and subsequent presidency. Furthermore, most current studies of fake news have tended to focus on the US context, where institutional trust levels in media and in the government are said to be at an all-time low (Edelman, 2018) and political polarization stands at an all-time high (boyd, 2017). In that context, it has been noted that the standard designation of “fakeness”, as a diagnosis to be remedied by “fact-checking”, fails to acknowledge a much more profound epistemological problem. As has long been argued in the literature on the sociology of scientific knowledge, “facts” are better understood as products of negotiated settlements amongst domain experts (Latour and Woolgar, 1976). The atmosphere of general suspicion towards expertise that underpins the rise of national populism thus poses a fundamental epistemological problem. This same general atmosphere of suspicion furthermore works to undermine trust in professional media institutions as the arbiters of facts. It is argued that this particular context plays into an innate psychological tendency to seek out biasconfirming information. 47

A leading scholar in the field recently posed the dilemma thusly: in the US, somewhere between “25 and 30 percent of Americans willingly and intentionally pay attention to media outlets that consistently tell that audience what it wants to hear, and what that audience wants to hear is often untrue” (Benkler et al., 2018: 367, emphasis added). In the aforementioned context, such scholars furthermore suggest that technocratic solutions designed to regulate and censor this fake news would be “neither feasible nor normatively attractive as they would certainly generate heated protest from a large spectrum of the populace” (367). Even in less politically polarized contexts the problem of regulation is extremely challenging. It is not isolated cases of fake news that are at issue but the larger problem of what these scholars refer to as “network propaganda”, which constructs “materially misleading” narratives from a tissue of facts (102). Because it is extremely difficult to establish “ground truth”, reliable technological solutions to the problem of fake news are thus unlikely at present (377).

In light of the former diagnosis, the empirical study below reframes the issue of “fake news” in the Dutch-language YouTube space by profiling the emergence of a network of channels engaged in political debate and commentary. It conceptualises elements of this network as hyperpartisan, in the sense that they are “openly ideological web operations” (Hermann, 2016).

Whilst marginal in comparison to mainstream Dutch news organizations these channels nevertheless appear highly engaging, at least from the perspective of the YouTube algorithm. As alternative news organizations almost all of these channels are unique to YouTube, making them “natively digital objects” (Rogers, 2013: 1). The empirical research that follows is thus concerned

47 Indeed from the social psychology perspective, “fake news” would arguably represent a more “natural” human preference than “facts”, insofar as the former more readily provides support that conforms to the “moral foundations theory” of human values (see: Haidt 2012).

 154 with understanding how these channels work, what their issues are, how they “do” Dutch national politics, and how they differ from the mainstream.

Figure 1. Related channels on YouTube. Table where the top row displays the name of each Dutch political party and the columns below each of these are the media organizations associated with each party’s YouTube channel.

29 March 2019. 48

YouTube’s “related channels” and Dutch political space

Following the “digital methods” approach (Rogers 2013), the discussion that proceeds here can be considered as an endeavour to “repurpose” YouTube as a research device by thinking along those lines that the platform makes available the public. In particular the approach uses

YouTube’s “related channel” algorithm as the basis for an analytical method that takes a set of

Dutch alternative news channels as its primary site of study. As a forewarning, it is important to recognize the contrived or “artificial conditions” with which the medium frames the object

(Rieder et al., 2016: 3). These conditions effectively make it impossible for the digital methods researcher to identify where the medium ends and where in turn the social begins. Though we do have a sense of how some of YouTube algorithms work from both the official corporate statement (Press, 2019), as well as from attempts by scholars to “reverse engineer” or

“teardown” the platform (Bessi et al., 2016), the precise functioning is unknown and in any case likely to change, thus frustrating the exact reproducibility of any of our findings. At any time,

YouTube may furthermore suddenly and unaccountably change its algorithms, which are in any case invisible to all but certain engineers at YouTube. Needless to say, the capriciousness of platforms renders the effective control of variables practically impossible. Whist the latter is axiomatic to digital methods it should also be recognized as an inherent limitation of the methods as well. For

48 Note that the Dutch labour party visualized on the far right of the graph did not return any related channels.

 155 these reasons the present report is thus best approached as “snapshots” of a milieu that is constantly in flux.

The empirical research focuses primarily on repurposing YouTube’s “related channels” for the purpose of analysis of the Dutch political space. In order to delineate what we are here calling the Dutch “political debate space” in YouTube, we started from the channels corresponding to the Dutch political parties. Since all 13 Dutch national political parties currently in the parliament have a YouTube presence, we used these channels as a “seed list”, or set of starting points, for the subsequent research. Starting then from this seed list the first technique compares all of the channels that YouTube classified as related to those of the Dutch political parties. This particular approach to categorization in all likelihood involved no human oversight; rather, it should be understood as an artefact of how the algorithm “values” the object, in relation to the aforementioned “engagement” metrics. Following the digital methods approach, the analytical gambit here is that the channels that YouTube’s suggests may be treated as a measure of how the platform views those parties. 49

The most unusual finding is that the algorithm relates one particular channel to almost all parties across the political spectrum: Forum voor Democratie (FvD). As a new “Eurosceptic” party with a younger demographic than the established nationalist populist Partij Voor de Vrijheid

(PVV), FvD and its agenda seem to dominate discussion in political debate in a network of

“alternative” channels discussed below, several of which YouTube relates to the parties, most notably “TheLvkrijger”. Before going on to discuss these alternative news channels in detail, the next most striking finding here is how the algorithm seems to organize the political spectrum in relation to different “establishment” news organizations. One cluster of parties is associated with CNN, ABC, NOS 50 and another around De Telegraaf, media organizations that may be considered as relative liberal and conservative/populist, respectively. While it is not necessarily easy to arrange the Dutch political parties on a left-right axis — as many smaller parties are more issuebased — it is worthy to note how the algorithm groups the new Groen Links and Denk parties with center-right and right-of-center parties. In addition to De Telegraaf, the algorithm also relates all of the parties in this latter cluster to alternative Dutch news organization: Omroep PowNed, a public radio and TV broadcast renowned for its satirical news show, PowNews, which often ridicules politicians with provocative questions. In what follows we will categorize Omroep

PowNed, along with GeenStijl a blog popular for its similarly abrasive style, as members of the established anti-establishment alternative news organizations.

49 One should note here that social media use machine-learning for “predictive models of consumers”

(Kitchen/Dodge) in which “success” is a measure of how correctly the algorithm predicts what a user will engage with. A well-known critique here is the notion of the “filter bubble” (Pariser 2011), which argues that algorithmic categorization can have the effect of narrowing the range of alternate viewpoints that one is exposed to.

50 Note that we removed most US channels from Dutch media network visualization below.

 156 Figure 2. TheLvkrijger post: Translated in English to: “He who is silent agrees! Don’t shut up anymore! This is your country! Claim it”.

That the algorithm also relates the parties to a smattering of large Dutch commercial and public media channels (WNL, RTL Nieuws, NPO Radio 1, Veronica Inside), is unsurprising as these would be an expected part of an average Dutch media diet. What is likely surprising to those unfamiliar with the Dutch political space in YouTube is the network of alternative or “alt” channels that YouTube relates to the parties, notably the aforementioned “TheLvkrijger”, but also “ARNews”, “Luekste YouTube fragmenten”, “Lissauer” and “Rafiek de Bruin.” With the possible exception of ARNews, all of these channels could be categorized as “openly ideological web operations”. As we will see, these Dutch political debate channels are “natively digital objects… ‘born’ in the new medium” (Rogers, 2013: 19), as opposed, for example to Omroep

PowNed. While some of these channels, like TheLvkrijger, are transparently partisan, national populist sentiments seem common in this space, as for example captured in a post by

TheLvkrijger encouraging viewers to vote in the upcoming elections, which featured the slogan “He who is silent agrees! This is your country! Claim it”.

 157 Figure 3. Related channels on YouTube. Panoramic graph of larger Dutch YouTube media sphere. This graph was reproduced twice two months apart with identical outcomes on 29 March 2019 and again on 22 May 2019.

The Dutch YouTube media sphere

In an effort to create a panoramic graph of the larger Dutch YouTube media sphere that would also remain connected to the Dutch political sphere on the platform we used YouTube’s related channels algorithm to “snowball” out from the seed list of the 13 parties to 3 degrees of relations. We subsequently visualised the related channel network with network analysis software, where nodes represent channels and edges represent relations according to YouTube’s algorithm. The relative size of the text represents a measure of their relative importance within the network. Finally, relative similarity between channels determines their colouration, clusters which we have then labelled as government, military, commercial, vlog, public, sport and finally our specific object of study. The largest nodes in the graph are all “establishment” media organizations with NOS Jeugdjournaal, RTL Nieuws and De Telegraaf at the center. Slightly outside of the center another large node is the established, anti-establishment channel Omroep PowNed,

 158 known once again for its “edgy” confrontational style of reportage. If one continues along this same line one encounters the cluster labelled alternative media ecology at the center of which the most connected node is FvD but which also includes a few government channels (for example

Eerste Kamer) as well as a number of the aforementioned “alt” political debate channels which we encountered earlier (for example “Leukste YouTube Fragmenten”). In the next steps of the analysis we will delve more deeply into these “alt” debate channels channels by performing some qualitative analyses of their content.

In both the panoramic map as well as in the prior analysis (based on only a single degree of relations to the seed list), we find the presence of a number of “natively digital” political debate channels, such as Leukste YouTube Fragmenten and TheLvkrijger. In considering these channels as a type of mini-genre, we can thus compare their style and how they “do” Dutch politics. At the outset it should be noted that, by certain measures, some of these channels appear quite marginal. TheLvkrijger, for example, which YouTube related to half the parties, only has 6.5 thousand subscribers. CNN, which YouTube also related to half the parties, has 6.5 million subscribers. The Dutch political space on YouTube is not that large, however, and in any case, despite differing by orders of magnitude, YouTube related channels algorithm places CNN and TheLvkrijger on the same footing. One degree of relations gives us a collection of “alt” political debate channels including 'TheLvkrijger’, ‘Leukste YouTube Fragmenten’, ‘Rafiek de

Bruin’, ‘LISSAUER.COM’, ‘Res Cogitans’, ’Omroep PowNed’, ‘ARNews’, to which we can add a few more by exploring their relations including ‘GeenStijl’, ‘AllePolitiek’ and

‘Deweycheatumnhowe’. In analyzing their style we can observe that ‘TheLvkrijger’, ‘Leukste

YouTube Fragmenten’ , ‘Rafiek de Bruin’, ‘AllePolitiek’ and ‘Deweycheatumnhowe’ are all of a sort, in that all post debate clips or interviews. Furthermore, sites as ‘ARnews’ and ‘LISSAUER’ use “meme” graphics — a style also employed, and in fact pioneered to an extent, by PowNed and GeenStijl. Somewhat like Omroep PowNed in style, GeenStijl is famed for its provocative anti PC tone. Settled in the Dutch media landscape (and with PowNed receiving structural funds from the government), they can thus fairly be labelled as “established anti-establishment”. Using clickbait tactics to attract attention, with the notable exception of AllePolitiek, the aim of these channels seems to be to amplify dissensus in the Dutch political space. Whilst this of course stands in marked contrast to the country’s long history of consensus politics, where one normatively stands on this depends on one’s democratic political theory. Furthermore, whilst several channels are transparently partisan, what it remarkable is that the majority of the most viewed videos in most of the channels focus on figures from the FvD and PVV.

To provide a synoptic view of the natively digital debate channels’ issues one can look at the most commonly used words in the titles of all of the channels in the form of “word clouds” with words colour-coded and sized by frequency. Those appearing in black are issues such as referendum, climate agreement, dividend tax and Brexit, whist those in colour are the names of parties and their spokespeople. At first glance, what one notices is that ARNews and AllePoeliek appear primarily issue driven, whilst the other channels seem more engaged with Dutch political personalities. One can also observe the relative similarity between ResCogitans and Leukste

YouTube Fragmenten, as channels that both appear partisan towards FvD — on closer inspection this is indeed the case (and in fact they even appear to be run by the same person).

Similarly, TheLvkrijger appears to be partisan towards the PVV, which is also the case on closer

 159 inspection. As with the thumbnails, discussed above, the names of the figures from both these parties commonly appear in all these channels video titles. Further scrutiny reveals all of these channels to be at least somewhat sensationalistic, with ARnews, often using terms like heated debate (“verhit debat”) in order to describe content. The more partisan of the channels follow an antagonistic logic when commenting on parliamentary debates, identifying the winner or loser of a given debate, at times resembling a debate genre familiar on YouTube, for example in videos featuring Jordan Peterson, often labelled in the style: Jordan Peterson DESTROYS so and so.

Figure 4. Thumbnail diagram of the ‘fringe channels’’ top ten most popular videos.

Alongside the related channels findings, the fact that official Dutch parliament channels, along with Forum voor Democratie (but not the other Dutch political parties), seemed to clustered alongside these “alt” debate channels seems peculiar. Given the aforementioned capriciousness of platforms, might these findings be attributable to an excited algorithm in the aftermath of FvD’s surprising success in the senate elections? If so, then one would expect for these findings to differ when reproduced at another point in time, either revealing an underlying stable state of network composition or else another excited state. With this question in mind we reproduced these first two methods, that were initially explore at the time of the senate election, at the time of the EU election. Remarkably, we found no substantial difference in either the channels that

YouTube considered as related to the parties (see Appendix 1). Moreover, the panoramic graph remained identical, 51 suggesting that it may thus reflect an underlying stable state of how the algorithm currently categorizes the larger Dutch YouTube media sphere in YouTube (see Figure 3). Because the EU elections did involve several other parties, we did however identify the presence of two new clusters in the panoramic graph: one of which, associated with the new pan-European Volt party, floats on its own completely disconnected from the overall network; and another, associated with Dutch Pirate Party, which is connected to the larger network via a

51 YouTube disabled the related video feature shortly after we completed this analysis (YouTube, 2019).

 160 channel ‘talking-head chat show’ called ‘Cafe Weltschmertz’. In close proximity to the alternative media ecosystem discussed above, Cafe Weltschmertz seems to frame its political debates in a tendentious style similar to some of the channels profiled above — referring to its approach, for example, as “politically incorrect”. In this same cluster we also however find leftist investigative journalism channels including ‘Follow the Money’ and De Correspondent as well as the expected channels focused on the issue of privacy, ‘Bits of Freedom’, ‘Privacy First’ and finally a debate channel called ‘Potkaars Podcast’ featuring a video on its front page, entitled “Potkaars prat met iedereen”. In light of our subsequent discussion of fake news as a topic of debate, the video’s description is worth quoting at length: “If you want real news, you have to cut through the smoke -smokes & mirrors- to get to information and demand a controllable government. Putting information away as ‘fake news’ is easy. But what do you replace it with?”

Figure 5. Screenshot from the “About” page on Cafe Weltschmertz’s YouTube channel which includes a sarcastic “trigger warning” for viewers whom might be angered by its frank approach to political debate, as well as cryptonormative espousal of “democratic hygiene processes”. 52

On fake news as issue

The final analysis concerns how it is that channels in the Dutch political space “do” the one issue in particular: fake news. We begin with a video from TheLvkrijger of PVV representative Martin Bosma confronting the government minister of Internal Affairs, Kajsa Ollongren in a Tweede

Kamer debate on the fake news that became central to her portfolio. In the video Bosma accuses Ollongren of “playing a strange game” with “what is truthful and what is not”.

Bosma points to a fundamental lack of consensus of what’s at issue in the fake news controversy

52 Without offering any analysis of this particular unique term, for reasons of brevity and focus, it is nevertheless worth noting here that one of the signature accomplishments of some of the American alternative partisan news system, especially those on the far-right, has been to introduce new terminology in the hopes of normalizing certain formerly radical conceptual frames (Hatewatch Staff, 2015; Benkler et al., 2018: 128-132). In political punditry this technique is sometimes called “opening the Overton window” (Marwick and Lewis, 2017: 11)

 161 more generally as well as alleging that Ollongren has seemingly tended to change her own definition of what constitutes fake news in order to suit her political purposes. When examining the comment section below this video we see commenters echoing Bosma’s sentiments and questioning Ollongren’s integrity, expressing the need for a concrete definition of fake news (45 likes). Commenters furthermore speak of Dutch public broadcasting as fake new that does

‘nothing but mislead citizens’ (78 likes). 53 This latter use of the concept of fake news echoes

Trump’s use of the term as means of attacking the establishment media.

Another video of interest, also published by TheLvkrijger, features a PVV-organized populisttype debate with pundits on the topic of fake news (‘nepniews’) and the European Union. Similar to the aforementioned Trumpian framing of fake news, the debate discusses the supposedly leftwing bias in the establishment media, as represented in one participant’s statement that “media serve the ideology of the establishment”. Again we see positive reception on the comment section where a commenter writes about the Dutch public broadcaster “NOS = FAKE NEWS”, and advocates viewers to seek their news form alternative sources on YT. 54

In another video on the topic, this time published by GeenStijl, a reporter asks politicians leaving the Tweede Kamer about the issue of ‘fake news’. This time the reporter’s questioning revolves around proposed European legislation, rather than Ollongren’s engagement with the issue. As per the channel’s provocative style, the video does not hide its partisan stance on the issue, titling the video: “Brussels is censoring free speech”. Again representative Bosma appears, this time with an attack on liberal political correctness emanating from the liberal technocrats in Brussels, stating “everything that is not politically correct will be tackled”. 55 By contrast other politicians interviewed by the journalist see the necessity of government action in response to the ‘crisis’ of fake news. In the comments section multiple commenters again reiterate the theme of the Dutch Government itself being “fake news”.

A video published on Leukste YouTube Fragmenten features a Tweede Kamer debate fragment, once again on the concept of freedom of speech, this time by FvD leader Thierry Baudet. In this clip Baudet makes a sophisticated conceptual point on the alethiological (the study of truth).

Using logic, Baudet tries to refute Ollongren’s concept of fake news as fallacious. He argues that if for an atheist god is not true, then that would not make preaching a form of disinformation.

Based on this argument he then claims that Ollongren would “accuse the teachings of

Catholicism of being untrue” and thus “a form of disinformation”. After his sophistry, Baudet then goes on to make the point that state actors should not be allowed to decide what is true and what is not true. “You cannot trust the state”, he says, what “we need”, he argues is “free press”. In the comments section commenters state that all politicians, besides Baudet, define fake news

53 The number of likes on a comment can be treated here as a measure of agreement with these sentiments expressed therein.

54 This theme of framing of “NOS is fake news” and “NOS is left-wing propaganda” came up in multiple comments of multiple videos.

55 Political correctness is a very popular straw man amongst “dark intellectual web” figures like Jordan Peterson on the right (Weiss 2018), but also left-wing figures such as Slavoj Zizek.

 162 subjectively in particular falling back on the Russian “evil actor” narrative, which a commenter characterized as “Orwellian”.

Figure 6. Word Clouds of the titles of all the videos from the political commentary channels.

 163 Figure 7. Screenshot of a comment under the video of ‘Leukste YT Fragmenten’, referring to a ‘hopeless debate’ and the lack of consensus on the definition of ‘nepnieuws’.

Although our analysis in the report did not include any left-of-center Dutch political commentators, this is not to say that they do not exist on YouTube, merely that the methods we used did not bring them to the fore. Indeed, alongside the “alt” channels profiled above we can in fact find a video of Arjen Lubach’s Zondag met Lubach, the VPRO broadcast in which the commentator, as with the one on the Green Style video, critiques the Russian “evil actor” narrative. In Lubach’s opinion the real threat is in fact an alt-right conspiracy theory, in the style of Pizzagate, which Russian actors merely amplify.

Conclusions: Left-leaning bias?

Academics are often accused by conservatives of having a left-leaning bias; indeed, apparently evidence reflects these allegations (Abrams, 2016). This narrative of “liberal bias” has been one of the central themes of the American new right, extending from contemporary “neoreactionaries” (Malice 2019), to 1990’s “culture warriors” (Nagle 2017), and back to the 1960’s

“messengers of the right”, who pioneered new media formats in order to disseminate their message (Hemmer 2016). And whilst accusations of such perceived liberal bias may be offered against this report, the fact remains that we came by the data underlying our findings by merely following the platform and the way that it categorized the Dutch political parties. In doing so we identified a series of “alternative” debate channels many of which appeared hyperpartisan — following Hermann’s initial definition of the concept as “openly ideological web operations”. If we were to locate the political bias of these “alternative” political debate channels in relation to

“establishment” media organizations in the Netherlands, then many would seem to be roughly aligned with the conservative and populist tone of De Telgraaf. Closer still to the antagonistic debate style that we observed in many of these channels is the transgressive style of reportage pioneered by the “established anti-establishment” of GeenStijl and Omroep PowNed.

The Netherlands is also well known for having innovated new new media formats, notably reality TV. Additionally, one might also say that the Netherlands has been innovative in developing new positions and issues on the right — notably the issue of homo-nationalism (Aydemir 2011).

What we may however also be seeing in this research is the possible emergence of US-style rightwing punditry in the Dutch sphere. While it still seems marginal in the current “alternative” debate space on YouTube space, exemplary here is the channel of ‘Paul Nielsen’ (24,531 subscriptions), an English language Dutch “alt-lite” channel which features such titles as: “NOS is the Dutch CNN | Biased News in Holland” and “How Marxists took over the Netherlands”.

 164 The site claims to be endorsed by Prof. Dr. Paul Cliteur, defendant at Wilders trial and PhD supervisor to Thierry Baudet. This channel may be a bridging node to the figures in what has been called YouTube’s “dark intellectual web” (Weiss 2018) or its “alternative influence network” (Lewis 2018), such as for example Stephan Molyneux who features a video with the title: “The Truth About Immigration and Crime in the NL”. At the same time, in scrutinizing a network one should be careful of the guilt by association fallacy. The point is rather to acknowledge the proximity to an active and controversial area of debate within the platform.

While the possible intervention of “Russian trolls” as a factor in 2016 US elections has been convincingly made (Jamieson 2018), the Dutch case is different. In addition to the absence of an Anglo-American “first past post” electoral system there is a very different media ecosystem in the Netherlands, which, for example still has a much higher trust in the general “establishment” than in the US (Edelman 2018). Furthermore, as opposed to the “neutrality” axiom that has characterize 20 th -century US news media, Dutch news media have always been partisan. This having been said what we see in YouTube suggests the emergence of a hyperpartisan Dutch new media political space. Currently this is mostly dominated by one party, but other parties may take this as a challenge. Insofar as YouTube represents a media source in the Netherlands, especially for youth, the Dutch YouTube “alt” political debate space may represent a re-politicization of youth, which runs counter to neoliberalism’s historical project of pre-emptive depoliticization

(Foucault 2008). If political pluralism advocates peaceful coexistence of different interests the combative and anti-politically correct tone of much of political debate on YouTube may militate against this. Can the long tradition of consensus in Dutch culture be brought to bear on this new debate culture or is the Netherlands on the path to Americanized Trump-style polarization? In terms of final takeaways, we can say that an inquiry into fake news, which defines the latter as the deliberate manipulations of facts, must also consider the inherently problematic aspects of this very conception as well. For this reason, regulating disinformation can be portrayed as Orwellian “thought control” which in turn resonates with populists’ anti-establishment, conspiratorial frameworks.

 165 Appendix 1

Related channels on YouTube, 22 May 2019. Table where the top row displays the name of each Dutch political party who ran candidates in the EU election. As with figure 1, the columns below each of these are the media organizations associated with each party’s YouTube channel. The related channels for the parties are identical to figure 1 apart from a few minor differences and the fact that D66 now no longer returns any related channels, as with PvdA. Note also that of the two EU parties that return channels are categorized quite differently than the other national

Dutch political parties.

 166

CONCLUSIONS: MAINSTREAM UNDER FIRE

Richard Rogers and Sabine Niederer

Conclusions: Mainstream under fire

Richard Rogers and Sabine Niederer

Separating disinformation and fake news and developing other notions further

Disinformation and fake news are contemporary phenomena with rich histories. Disinformation, or the willful introduction of false information for the purposes of causing harm, recalls infamous foreign interference operations in national media systems, such as the Russian campaign ‘Operation Infektion’ that in the early 1980s effectively publicly linked the HIV virus with a supposed, secret US bioweapons lab. Outcries over fake news, or dubious stories that have the trappings of news, have occurred repeatedly with the introduction of new media technologies that disrupt the publication, distribution and consumption of news – from the socalled rumour-mongering broadsheets centuries ago to the blogosphere more recently. Social media are only the most recent ‘truthless’ media. Designating a news organization as fake, or calling it der Lügenpresse, however, has a darker history, associated with authoritarian regimes or populist bombast diminishing the reputation of ‘elite media’ and the value of inconvenient truths more generally.

These days social media platforms have been implicated in both the practice of disinformation as well as the rise of these two varieties of fake news. As discussed in the theoretical and empirical scholarship to date, social media have enabled the penetration of foreign disinformation operations, the widespread publication and spread of dubious content as well as extreme commentators with considerable followings attacking mainstream media as fake.

Worldwide, disinformation and fake news are increasingly under study together, but the argument could be made to separate them. Indeed, in the Netherlands evidence of foreign disinformation campaigning is scant; cases of domestic actors employing the ‘Russian playbook’ of disinformation tactics are also seldom documented. Unlike in the case of the US, to which much of the scholarship is dedicated, in the Netherlands one has not witnessed the rise of imposter news organisations or the formation of advocacy groups purporting to represent social groups or causes. Indeed, when employing narrow definitions of disinformation and fake news, there is hardly any to be found in the Netherlands.

But definitions of fake and its next-of-kin ‘junk’ news often extend to clickbait, conspiracy, hyperpartisan and tendentious sources as well as artificially amplified social media content and accounts. As a case in point, when Buzzfeed News famously reported in 2016 that ‘fake news’ was outperforming mainstream news on Facebook, included in its definition were clickbait and hyperpartisan sources, such as Breitbart News. Expanding the definition in such a manner would have consequences in that the Netherlands has all of them in relative abundance.

Initial studies have found that the Dutch are great consumers of clickbait and ‘pulp’ content; there is a well engaged-with set of tendentious and highly partisan news-like organisations especially on the right of the political spectrum, and the artificial amplification of social media accounts, including those of certain politicians and musicians, has been well documented. Their sway varies. Clickbait is said to be consumed more often than mainstream news, though there is

 167 also more of it. Conspiracy is perhaps the least clicked on, at least according to our findings per platform, discussed below. In political spaces online, news furnished by commercial and public broadcasting are still referenced, liked or shared in greater quantities than tendentious and hyperpartisan sources, though the latter has been present in the most engaged-with lists of sources around election issues. Artificial amplification both burnishes one’s image but also has led to mini-scandals when fake followers are revealed through new online detection tools and news reporting.

Whether any of them is particularly persuasive is a question increasingly posed. The first wave of scholarship on the production and spread of disinformation has yielded to the next wave on its effects. Given people’s hardened attitudes the question concerns whether the influence of disinformation and fake news is ever more than ‘minimal’.

In that regard, the rise of extreme content (including extreme clickbait), circulated on social media platforms, is one source of continuing consternation and measurement, leading to calls for platform regulation and prompting social media companies to hire more content reviewers and work on automated detection. Another source of concern is the mainstreaming of doubt and trust in public institutions and media, concomitant with the rise of both ‘alternative facts’ and

‘alternative fact infrastructures’. The post-truth condition, as it is termed, is discussed as both first-order ‘fact fights’ as well as second-order competitions between ‘sectarian knowledge’ regimes and competing media ecologies. Is the authority of mainstream news and knowledge institutions declining for increasing segments of society that consume the alternatives? One finding often related is that older consumers are ‘available audiences’ for fringe media and are relatively ‘heavy users’.

The consuming and sharing of fake news have been the subject of media literacy initiatives, including quizzes, serious games and public service campaigns. Through heightened awareness, especially around the time of elections, the impact on consumers of any disinformation and dubious content may be mitigated and the institutions made resilient, it has been argued.

Voluntary and professional fact-checking are also discussed in this regard, as are (automated) content flagging, together with the need for human review. The question regularly posed is whether the sheer amount of platform junk will overwhelm the capacity to review it, together with the related issue of who will review the reviewers.

Finally, there is widespread scholarly concern about the restrictions to public access of social media data, otherwise known as the issue of ‘locked platforms’. Future research should address the extent to which disinformation and fake news (in all its definitions) continue to thrive online, and whether there are monitoring capacities in place so that its overall consumption and persuasive capacity may be measured and the wider societal implications may be studied and acted upon.

Empirical findings concerning junk news around the Dutch elections of 2019

The present study consists of a series of empirical case studies concerning the engagement with fake/junk news, together with hyperpartisan and tendentious sources, in Dutch political spaces in social media in the run up to two elections in 2019. These spaces were demarcated using

 168 queries of politicians’ and political party names as well as social issues, some related to the elections (such as climate and EU) and others more controversial (such as Zwarte Piet). Here the findings are summarised, and subsequently put into perspective in a discussion of their implications, also for policy.

The overall research question driving the study is as follows. To what extent do disinformation and so-called fake or junk news resonate in political spaces online within social media (and search engine returns) around the 2019 provincial elections and the European parliamentary elections in the Netherlands?

Here the findings are provided in brief.

• 1) 

  We found neither foreign disinformation (campaigning) nor fake advocacy groups operating around the Dutch provincial and European parliamentary elections of 2019.

• 2) 

  Mainstream news is consumed and engaged-with more than junk news, but not for all platform in all periods under study (8 February - 25 March 2019, 26 April - 24 May 2019 or longer durations). The issue spaces around Zwarte Piet and MH17 have proportionately higher quantities of junks news than election issues and are also ‘year-round’ issues, so to speak.

• 3) 

  With respect to social media manipulation, troll-like users are active across Dutch political issues spaces (on Twitter). We also suspect there is artificial amplification taking place (fake followers on Instagram).

• 4) 

  There is the emergence of a hyperpartisan/tendentious right-wing (separate) media space, competing with the mainstream news and also mainstreaming, in the sense that these sources are spread by regular (non-suspect) social media users as well as troll-like users.

• 5) 

  Proportionately, Facebook has the greatest amount of fake/junk news compared to other platforms, followed by Twitter. YouTube is a significant right-wing news space generally, and debate space for ‘fake news’ as issue.

• 6) 

  Dutch 4chan and Reddit circulate far more Dutch mainstream than junk news, with 4chan users likely commenting upon rather than taking over its narratives. 4chan is an incubator of farright activity in the Netherlands as seen through the types of YouTube videos referenced.

  Facebook: Fertile ground for junk news The method behind the research presented here derives from data journalism, particularly that of Buzzfeed News, and later the NRC Handelsblad, which both ascertained the most engaged-with stories on Facebook in the run-up to national elections. Whereas Buzzfeed News found that ‘fake news’ on Facebook outperformed mainstream news in the months preceding the US federal elections of 2016, leading in part to the overall ‘fake news crisis’, the NRC Handelsblad, deploying a far stricter definition, found scant presence of such material prior to the 2017 Dutch national elections. Our study found that Facebook is a fertile ground not for disinformation and fake news in the Netherlands but rather for junk news, a roomier definition than fake news as

   169 discussed above. Whilst it is not outperforming mainstream news, it is far from marginal, and in a few periods under study its presence in the top stories on Facebook (judged from engagement measures) equals or outperforms the mainstream. Engagement, measured by such interaction as shares and likes, requires further study, however, for it should not be equated in each case with support or agreement.

While Facebook contains a significant stream of junk news, including hyperpartisan, tendentious and conspiracy sources, foreign disinformation and fake news – such as organisations pretending to be news sources or advocacy groups – were found to be absent, at least for the most engagedwith stories related to the elections gathered through keyword queries of political leaders, parties and social issues (or in longer longitudinal studies of certain issue spaces on Twitter as well as in Dutch spaces on Reddit and 4chan). Disinformation and fake news may have not been encountered, but junk news is a factor in Dutch political spaces on social media and its impact should be critically studied.

Google Web Search: vernacular search queries result in junk news The Google Web Search study involved locating fake/junk news within the top twenty results in Google.nl for queries of Dutch political parties and significant social issues prior to the Dutch provincial and European parliamentary elections of 2019. The queries were formulated by combining the names of the political parties with social issue keywords. The keywords derive both from the ‘official’ issue language collected from the party platforms as well as vernacular terms distilled from the comments on political party Facebook pages. Of the junk news found nearly all originate from hyperpartisan and tendentious sources rather than disinformation, conspiracy theory or clickbait. For particular groups of issue queries, up to 25% of the results were hyperpartisan. As on the other platforms under study, in the ‘top content’ no foreign disinformation, fake news organisations or fake advocacy groups were present during the election periods.

The social issue keyword queries in combination with right-of-center political parties resulted in junk news sources in greater quantities than that of left-of-center party names. It was also found that the presence of junk news is not stable over time. Prior to the provincial elections the quantities rose, only to decline the day of the election and in its immediate aftermath, as witnessed by the issue of migration. The inverse was witnessed during the European parliamentary elections. When comparing the two types of search queries, the official and the vernacular, the latter results in a higher percentage of junk news in the results.

Twitter: junk news and troll activity around polarising subject matters The Twitter study examined the presence of junk news, tendentious news as well as troll-like activity during the campaign periods around the Dutch provincial and European parliamentary elections of 2019. There was troll-like activity encountered around the provincial elections around political terms such as the tag for the elections themselves (‘PS2019’), certain party leaders as well as potentially polarising issues such as MH17, Zwarte Piet and the Utrecht tram shooting of March 2019. Troll-like activity refers to a series of behavioural indicators, including targeting politicians with unusually high bursts of tweets in short period of time or through a set of accounts created at about the same time. The analysis found fourteen troll-like Twitter users

 170 were active around all issues studied and twenty-nine around most of them. Four of these profiles remained active (or became active again) around the European parliamentary elections.

They all spread mainly hyperpartisan and tendentious sources, followed in quantity by conspiracy websites.

In answering the question of the extent to which junk news is present in the Dutch political

Twittersphere, we ascertained the most shared sources, finding a steady resonance of junk news, paling in comparison however to mainstream sources. One finding of note is that during the

Provincial elections both Zwarte Piet and MH17 witnessed proportionately high amounts of activity, in spite of the fact that the Santa Klaus holiday (where Zwarte Piets make their appearance) does not take place until December and there was no particular MH17 news, for example concerning the investigations into the cause of the crash of the airliner. Troll-like users shared mostly hyperpartisan and tendentious sources, followed by conspiracy websites putting forward theories concerning MH17 and the Utrecht tram shooting. The pro-Russian site, novini.nl, which on a story level oscillates between hyperpartisan and conspiracy, also was circulated by troll-like users for all social issues under study, but only rarely in relation to political leaders. During the European Parliamentary elections junk news occasionally resonated more than mainstream news around such polarising issues as Zwarte Piet and MH17. It outperformed mainstream news largely owing to the lack of news coverage of these issues during the periods under study, when junk news remains steady.

Based on the findings, it appears that the Dutch Twittersphere does not have a disinformation problem; no professional or large-scale disinformation or fake advocacy campaigns were encountered. Troll-like users, whether in the form of bots or semi-automated users autoretweeting and posting original content, do lend some symbolic power to divisive points of view around several social issues. Polarising issue activity, fuelled by reference to hyperpartisan and tendentious news, remain rather stable (albeit marginal) throughout both periods of study, suggesting that these issues do not resonate at expected times of the year only, but rather throughout.

Instagram: a separate far-right media ecology and signs of artificial amplification The Instagram study inquired into the presence (and absence) of disinformation and fake/junk news in three ways: on a post-level, a source-level and that of followers. On a post-level, we examine the circulation of fake/junk news in political spaces, on a source-level we compare the audiences of fake/junk news and political leaders and parties, and finally we study the follower bases of the political entities, searching for signs of inauthentic behaviour. In all we found a relatively healthy Dutch political arena on Instagram with only small amounts of junk news and fake followers. The vast majority of liked content in Dutch political Instagram, demarcated through political keyword queries, is not junk news, though around certain right-wing political leaders and divisive issues small amounts of tendentious and hyperpartisan news appear. Mainstream news was more prominent than junk news in the posts related to political parties and leaders in both periods under study. The most active users of the platform in the Dutch political Instagram arena are seemingly authentic with little sign artificial manipulation.

 171 Within this relatively healthy political space online the only suspicious activity encountered was on the far right of the political spectrum, where the circulation of junk news takes place and where those who follow the far-right parties and leaders also follow the junk news sources. Here there are also indications of artificial manipulation. Certain right-wing leaders (as well as the personal account of the Prime Minister) show signs of a significant fake follower base. The artificial activity found is in line with the 2015 fake follower incident when Twitter announced the deletion of fake followers that affected certain Dutch politicians (and celebrities) disproportionately.

Reddit and 4chan: YouTube videos as news source contribute to polarisation Despite their characterisations as alternative spaces on the web, the ‘Dutch’ Reddit and 4chan, following from our findings, do not appear to spread alternative news sources, but rather refer more often (even overwhelmingly) to mainstream news sources. Apart from witnessing two examples of circulating the pro-Russian site, novini.nl, and the activity of one particularly suspect Reddit account, there does not appear to be any coordinated disinformation or fake news campaigning. There is certainly junk news to be found but compared to the overall spreading of sources the proportion is marginal. Particularly Reddit seems to be ‘resistant’ to disinformation. In 4chan the research did find different types of junk news, especially of the hyperpartisan variety, particularly in 4chan/pol/.

The research found the presence of junk news, especially of the hyperpartisan variety, particularly in 4chan/pol/. These are largely links to tendentious and hyperpartisan sources such as The Post Online and De Dagelijkse Standaard, but also to the conspiracy site, NineForNews. A right-wing orientation was to be expected in 4chan/pol/, given its reputation as a hub of the extreme right, but it was perhaps less likely for Reddit, as the ‘Dutch’ Reddit that we took as a starting point contains a number of politically diverse subreddits. The articles that have gained the greatest salience concern migration and Islam rather than such geopolitical content as the Russian involvement in MH17. Whilst it may be problematic to label these sources as ‘fake’, they could be characterised as polarising.

Nevertheless, the vast majority of the links to news sources are directed at mainstream outlets such as NOS.nl, both on 4chan/pol as well as Reddit. These findings are counter-intuitive in that the platforms are often described as alternative, as was said, and the anonymous and pseudonymous users point to marginal or alternative knowledge sources such as alternatives to Wikipedia. Our findings dispute such a characterisation, for mainstream sources enjoy some authority on these platforms, but we did not as of yet research how they are discussed, e.g., as the starting point for a discussion or ridicule. On Reddit it could be that the mainstream NOS.nl is considered a reliable source and on 4chan/pol/ “fake news”.

Finally, it important not to regard mainstream and junk news as the only sources of news on the web. In both the Reddit and 4chan research but also in the YouTube study, we found that YouTube has emerged as a major news source. That can be said particularly for 4chan/pol/ but also for Reddit, where “News & Politics” videos are a significant source. On 4chan/pol/ they may be the most significant, quantitatively leaving other sources well behind. From a small explorative study of the YouTube channels posted on Reddit we found established sources

 172 referenced such as PowNed, Zondag met Lubach and NOS, while on 4chan/pol/ alternative, foreign and political (hyperpartisan) sources are pointed to, such as Mike Cernovich and Russia

Today. Should such linking and engagement continue, such polarising content could have a polarising effect in the Dutch political space.

From findings to implications: Mainstream under fire

There is a small, but growing literature concerning how fake news could be considered a moral panic (Morozov, 2017; Hirst, 2017). The term refers to recurring episodes in history when “rightthinking people” (defined seminally by Stanley Cohen as “editors, bishops and politicians”) spot a condition that is supposedly prompting a decline in societal standards and values (1972). When fake news is viewed through that lens, the concern is about how traditional journalism as a pillar or ‘4th estate’ of democracy is being hollowed out by social media and replaced by low-quality clickbait as well as openly ideological commentary, both formatted in manners that drives their consumption not so unlike sugary junk food. The overall health of media as social fabric is said to be at stake, for citizens using social media as source for political information are disadvantaged in their capacity to form judgements about social issues and politics more broadly (Carlson, 2018). There is a second set of literature describing how the media coverage of fake or junk news, and especially its relationship to the growth of a right-wing media ecology, gives it

‘oxygen’ (Phillips, 2018). More poignantly, it has been argued that journalistic coverage should turn its attention to the victims, rather than to the fascinating subcultural milieu online where the far right cultivates itself. There are also cases of politicians’ forwarding extremist and divisive content, which also gives it oxygen in the sense that it contributes to its spread and perhaps to its normalcy. Along all these lines, the recommendations concern identifying and acting upon threats to the mainstream, be they from social media platforms or from within the professions and practices of journalism, online content creation and political leadership.

As we have found there are particular platforms and subject matters where the threats to the mainstream appear more acute. Whilst not a space where Dutch junk news sources are spread on a massive scale, the Dutch 4chan is an incubator of extremist sentiment, especially with respect to anti-semitism and anti-immigration. Other platforms are problematic for different reasons.

Dutch political spaces in Facebook and Twitter, demarcated through politician, party and issue queries, have the largest quantities of junk news that is engaged with, though they are still smaller than mainstream news consumption overall in those same spaces. Among the junk news, hyperpartisan sources (rather than disinformation or conspiracy) are amongst the more popular, and for divisive subject matters such as climate change, MH17 and Zwarte Piet their stories occasionally outperform those in the mainstream press. On Twitter during the European parliamentary election campaign period, for example, a pro-Nexit story in the hyperpartisan newspaper, De Dagelijkse Standaard, about the Netherlands leaving the EU outperformed a counterpart article in the mainstream NRC Handelsblad. A more general polarised media ecology is also in evidence. On YouTube an alternative (right-wing) media sphere has formed, where extreme YouTubers, or micro-celebrities, hold sway. Instagram also has a right-wing media space, analytically detected through shared followers of right-wing politicians and hyperpartisan media organisations. These are largely ‘alt lite’, meaning anti-establishment and anti-political correctness, with content that also could be considered anti-Islam. There are no discernable leftwing equivalents. Rather, these spaces compete with more mainstream ones.

 173 In contrast to the situation in other countries during the European Parliamentary elections, in our study we did not find foreign disinformation but rather so-called junk news, especially around particular issues, such as Zwarte Piet, MH17, climate and the European Union (Peel,

2019). We also found it around the topic of ‘fake news’, studied in this instance as a social issue. Although decent quantities of junk news were in evidence, mainstream news largely outperformed it. The largest quantities of junk news circulated not so much around political parties and leaders (with some exceptions), but around specific polarising issues. Junk news activity around these issues sometimes appeared during the election periods, but for other issues there was year-round activity, even for such seasonal issues as Zwarte Piet. Thus, the question is not only whether there is junk news around election time, but also more generally when it manifests itself, and with which intensity and duration.

The following policy implications of our work are directed specifically at the phenomenon of junk news, rather than at foreign disinformation and fake news from organisations feigning to be news organisations or fake advocacy groups, of which we found none, at least in the top or most engaged-with content related to Dutch politics across the web and social media platforms. Our recommendations concern the recognition and monitoring of the polarisation of the media landscape, the devitalisation or disengagement with extreme content, a national conversation about issues that appear frequently in junk news (such as Zwarte Piet) rather than one about disinformation or fake news generally, training for professionals that produce online content, and enabling access to the (increasingly inaccessible) data on social media platforms for research and media monitoring.

Policy themes in brief 1) The monitoring of the polarisation of the media landscape, and the mainstreaming of polarising media with extreme content on social media platforms.

Social media platforms rely on software, their users as well as content reviewers to detect extreme content. More and more of it is subsequently removed. But historically the attention paid by social media companies to extreme content has been uneven, and definitions unstable. It thereby remains desirable to institute independent monitoring. Such work could be taken up by academic researchers, non-governmental organisations, governmental agencies specialised in extremism and polarisation as well as media watchdogs.

• 2) 

  Media training for professional content makers – from journalists to digital media producers – concerning online source criticism as well as amplification or ‘oxygen-giving’ of extreme speech actors in society.

The Netherlands has existing media literacy training programs, designed for example for senior citizens as well as primary and secondary school students. This recommendation is made specifically for professional content-makers such as journalists and editors. It could be made a part of existing or new media literacy programs dedicated to online source criticism and dealing

 174 with polarising content (see also point three below). Such a training program is also of use to lecturers in higher education, policymakers and civil society.

• 3) 

  No oxygen-giving to extreme actors and their (online) content.

In our study we found that tendentious news stories circulate well during the election campaign periods and beyond. The articles are shared and liked by troll-like users but also by regular news consumers, which we found for example on Twitter during the Provincial elections. In the same spaces we also found users sharing and liking discriminatory, anti-Semitic, misogynist and xenophobic content, albeit it to a lesser degree.

It is important not to equate tendentious and extremist media, even when they appear to share standpoints without using the same words. Similarly, that tendentious media is on the rise and mainstreaming does not mean that similar weight should be given to extreme media, particularly right-wing extremist media. The recommendation is that no oxygen should be given to extreme media sources and their content, meaning no sharing, liking, reacting, commenting, retweeting or YouTube-debating. Any form of engagement with such content increases the attention and the metrics and contributes to its spread, ranking and normalisation. Such a recommendation goes for public broadcasting and commercial media organisations, but also for the tendentious media. Instead of journalists’ writing about far-right subcultures, attention could be spent on their victims (Philips, 2018).

• 4) 

  Recognition of polarising issues such as Zwarte Piet and the facilitation of national conversations.

The research found that attention to polarising issues such as Zwarte Piet is year-round rather than seasonal. Such recognition of increasing polarisation in society should lead to discussions about how common ground may be found. The Netherlands has a tradition of collective discussion concerning major societal issues through such mechanisms as the Brede

Maatschappelijke Debat (society-wide debate) and interactive policy making. There are other contemporary forms of citizen participation and discussion that could be instrumental in dealing with polarising issues and cultural contestation. Institutions experienced in organising societal discussion and debate should be called upon and supported to do so, and bottom-up initiatives should be facilitated.

• 5) 

  Advocacy for social media data access for researchers, journalists and watchdogs, and

creation of research archives of deleted content.

The current issue of ‘locked platforms’ concerns the extent to which social media companies are making their data inaccessible to researchers, journalists and non-governmental organisations. As an answer to governmental concern about ‘dark political posts’ (political ads directed only at a segment of users in their newsfeeds) and other political ads without clear provenance, Facebook has launched a political ad archive tool and API. But at the same time Facebook has removed in part or in whole access to services such as the Pages API and Graph Search, which had been in widespread use by researchers. Social media companies should take up the task of making

 175 available the data that researchers, journalists and non-governmental organisations would like to use for the purposes of research, monitoring and archiving. Governmental agencies, in consultation with the users and use types mentioned above, have a facilitative as well as a regulatory role to play here.

 176 References

Abbruzzese, Jason (2018) ‘Facebook became your news diet. Now, it’s going to serve you junk’,

Mashable, 17 January. Abrams, Sam (2016) ‘Professors Moved Left Since 1990s, Rest of Country Did Not’, Heterodox

Academy, January 9. https://heterodoxacademy.org/2016/01/09/professors-moved-left-butcountry-did-not/

Access Now, Civil Liberties Union For Europe, and European Digital Rights (2018) Informing the

“disinformation” debate, Report, Published online on October 18. https://edri.org/files/online_disinformation.pdf

Albright, Jonathan (2017) ‘Itemized Posts and Historical Engagement - 6 Now-Closed FB Pages’,

Data set, Tableau Public, 5 October, https://public.tableau.com/profile/d1gi#!/vizhome/FB4/TotalReachbyPage

_____ (2016) ‘The #Election2016 Micro-Propaganda Machine’, Medium, 18 November.

Alexander, Julia (2017) ‘Reddit’s New Policy Won’t Affect Some of Its Most Notorious, Hate-Filled Subreddits’, Polygon (blog). 2 November.

https://www.polygon.com/2017/11/2/16591508/reddit-content-policy-update-subreddit-banthe-donald-kia

Allcott, Hunt and Matthew Gentzkow (2017) ‘Social Media and Fake News in the 2016 Election’,

Journal of Economic Perspectives, 31(2): 211–236. https://doi.org/10.1257/jep.31.2.211 Alvarez, R. Michael, Ines Levin, Alexander H. Trechsel and Kristjan Vassil (2014) ‘Voting Advice

Applications: How Useful and for Whom?’, Journal of Information Technology & Politics, 11(1): 82- 101, DOI: 10.1080/19331681.2013.873361

Angwin, Julia, Madeleine Varner and Ariana Tobin (2017) ‘Facebook Enabled Advertisers to Reach “Jew Haters”’, ProPublica, 14 September. https://www.propublica.org/article/facebook-enabledadvertisers-to-reach-jew-haters

Aydemir, Murat (2011) ‘Dutch Homonationalism and Intersectionality’, ARC-GS Lecture. https://arcgs.uva.nl/videos/video-artikelen/dutch-homonationalism-andintersectionality.html?1556011447190

Bahara, Hassan, Annieke Kranenberg and Dimitri Tokmetzis (2019) ‘Leidt het Algoritme van

Youtube je Naar Extreme Content?’, De Volkskrant, February 11. https://www.volkskrant.nl/nieuws-achtergrond/leidt-het-algoritme-van-youtube-je-naarextreme-content~bea101e3/

Barbrook, Richard (1998) ‘The Hi-Tech Gift Economy’, First Monday, 3(12). https://ojphi.org/ojs/index.php/fm/article/view/631

Baumgartner, Jason (2018) ‘Pushshift API (version 1.0)’, API Documentation,

Pushshift. https://pushshift.io/api-parameters/ Beiler, Markus and Johanna Kiesler (2018) ‘“Lügenpresse! Lying press!” Is the Press Lying?’, in Otto,

Kim and Andreas Köhler (eds.) Trust in Media and Journalism, Wiesbaden: Springer, 155-179. Belk, Russell (2007) ‘Why Not Share Rather Than Own?’, The ANNALS of the American Academy of

Political and Social Science, 611: 126-140. Bell, Emily and Taylor Owen (2017) ‘The Platform Press’, Report, Columbia University: Tow Center

for Journalism. Bellingcat Investigation Team (2016) ‘Behind the Dutch Terror Threat Video: The St. Petersburg

“Troll Factory” Connection’, Bellingcat, 3 April.

 177 Benkler, Yochai, Robert Faris, Hal Roberts, and Ethan Zuckerman (2017) ‘Study: Breitbart-Led

Right-Wing Media Ecosystem Altered Broader Media Agenda’, Columbia Journalism Review, 3 March.

Benkler, Yochai, Robert Faris and Hal Roberts (2018) Network Propaganda: Manipulation,

Disinformation, and Radicalization in American Politics, Oxford: Oxford University Press. Berg, Kati Tusinski (2017). ‘Social Media, Hashtag Hijacking, and the Evolution of an Activist

Group Strategy.’ in Lucinda Austin and Yan Jin (eds.), Social media and crisis communication, London: Routledge, pp. 141-156

Bergstrom, Kelly. 2011. ‘‘Don’t Feed the Troll’: Shutting down Debate about Community

Expectations on Reddit.Com’, First Monday, 16(8). https://doi.org/10.5210/fm.v16i8.3498 Bessi, Alessandro and Emilio Ferrara (2016) ‘Social bots distort the 2016 US presidential election

online discussion’, First Monday, 21(11). https://firstmonday.org/article/view/7090/5653 Bessi, Alessandro, et al. (2016) ‘Users Polarization on Facebook and Youtube’, PLoS ONE, 11(8): 1–

• 13. 

  https://doi.org/10.1371/journal.pone.0159641 Boczkowski, Pablo (2016) ‘Fake news and the future of journalism’, NiemanLab, December.

https://www.niemanlab.org/2016/12/fake-news-and-the-future-of-journalism/ Boghardt, Thomas (2009) ‘Soviet Bloc Intelligence and Its AIDS Disinformation Campaign’, Studies

in Intelligence, 53(4): 1-24. Borra, Erik (2013). ‘DMI Tools’, wiki. https://wiki.digitalmethods.net/Dmi/ToolDatabase. Borra, Erik, & Bernhard Rieder (2014) ‘Programmed method: developing a toolset for capturing and

analyzing tweets’, Aslib Journal of Information Management, 66(3): 262-278. Borra, Erik, Sabine Niederer, Johannes Preuß, and Esther Weltevrede (2017) ‘Mapping troll-like

practices on Twitter’, in Liliana Bounegru, Jonathan Gray, Tomasso Venturini & Michele Mauri

(Eds.), A Field Guide to ‘Fake News’ and Other Information Disorders: A collection of recipes for those who

love to cook with digital methods, Amsterdam: Public Data Lab, pp. 161-196.

Bos, Menno van den (2018) ‘Nee, Jíj Bent Een Cultuurrelativist! Hoe Academische Begrippen

Strijdwapens Werden’, Vrij Nederland, 12 December. https://www.vn.nl/identiteitspolitiekcultuurmarxisme-intersectionaliteit/.

Bounegru, Liliana, Jonathan Gray, Tommaso Venturini, Michele Mauri (2018) A Field Guide to “Fake

News” and Other Information Disorders: A Collection of Recipes for Those Who Love to Cook with Digital

Methods, Amsterdam: Public Data Lab.

Bowden, Ian (2016) ‘How Google is tackling fake news, and why it should not do it alone,’ Search

Engine Land, 30 November, https://searchengineland.com/google-tackling-fake-news-not-alone

264058.

boyd, danah (2017) ‘Did Media Literacy Backfire?’, Points blog post, Data & Society, 5 January,

https://points.datasociety.net/did-media-literacy-backfire-7418c084d88d.

boyd, danah, Scott Golder and Gilad Lotan (2010) ‘Tweet, tweet, retweet: Conversational aspects of

retweeting on Twitter’, in 43rd Hawaii International Conference on System Sciences, Honolulu, HI:

IEEE, January, pp. 1-10, DOI: 10.1109/HICSS.2010.412.

Boyd, Ryan, Alexander Spangher, Adam Fourney, Besmira Nushi, Gireeja Ranade, James

Pennebaker and Eric Horvitz (2018) ‘Characterizing the Internet Research Agency’s Social Media

Operations During the 2016 U.S. Presidential Election Using Linguistic Analyses’, DataCite,

DOI:10.31234/osf.io/ajh2q.

 178 Bradshaw, Samantha (2018) ‘Responding to Fake News through regulation and automation’, in Fake

News, Authentic Views, Report, London: Carter- Ruck. https://www.carterruck.com/images/uploads/documents/RESPONDING_TO_FAKE_NEWS.pdf

Bradshaw, Samantha and Phillip N. Howard (2018) ‘Challenging Truth and Trust: A Global

Inventory of Organized Social Media Manipulation’, Computational Propaganda Data Memo, Oxford: Oxford Internet Institute.

Brattberg, Erik and Tim Mauer (2018) ‘Russian Election Interference: Europe’s Counter to Fake

News and Cyber Attacks’, White Paper, Washington, DC: Carnegie Endowment for International Peace, May.

Brennen, Bonnie (2017) ‘Making Sense of Lies, Deceptive Propaganda, and Fake News’, Journal of

Media Ethics, 32(3): 179-181. Broderick, Ryan and Jules Darmanin (2018) ‘The “Yellow Vest” Riots In France Are What Happens

When Facebook Gets Involved With Local News’, Buzzfeed News, 6 December. Bruns, Axel, Anja Bechmann, Jean Burgess et al. (2018) ‘Facebook shuts the gate after the horse has

bolted, and hurts real research in the process’, Internet Policy Review, 25 April. Bump, Philip (2017) ‘The investigation goes digital: Did someone point Russia to specific online

targets?’ Washington Post, 12 July. Burger, Peter (2016) ‘Moslims vernielen kerstmarkt in Litouwen? Nee: rellen in de VS’, De Gestolen

Grootmoeder blog, 14 December. http://www.gestolengrootmoeder.nl/wordpress/moslimsvernielen-kerstmarkt-in-litouwen-nee-rellen-in-de-vs/

Burger, Peter, Soeradj Kanhai, Alexander Pleijter, and Suzan Verberne (2019) ‘The Reach of

Commercially Motivated Junk News on Facebook’. ArXiv:1901.07290 [Cs], 22 January. http://arxiv.org/abs/1901.07290.

Butac, Maicolengel (2018) ‘The Reuters Institute for the Study of Journalism vs fake news’, BUTAC blog, 8 February.

Cadwalladr, Carole and Emma Graham-Harrison (2018) ‘Revealed: 50 million Facebook profiles harvested for Cambridge Analytica in major data breach’, The Guardian, 17 March.

Cadwalladr, Carole (2016) ‘Google, democracy and the truth about internet search’, The Guardian, 4 December.

Canadian Security Intelligence Service (2018) Who said what? The Security Challenges of Modern

Disinformation, World Watch: Expert Notes series publication (No. 2016-12-05), Canada: Canadian Security Intelligence Service.

Carlson, Matt (2018) ‘Fake news as an informational moral panic: the symbolic deviancy of social media during the 2016 US presidential election’, Information, Communication & Society, DOI:

10.1080/1369118X.2018.1505934, published online on August 1. Cary, Stephen G. (1955) Speak Truth to Power: A Quaker Search for an Alternative to Violence,

Philadelphia: American Friends Service Committee. Chadwick, Andrew (2013) The Hybrid Media System: Politics and Power. Oxford: Oxford University

Press. Chen, Adrian (2015) ‘The Agency’, The New York Times, 2 June. Claussen, Victor (2018) ‘Fighting hate speech and fake news. The Network Enforcement Act

(NetzDG) in Germany in the context of European legislation’, Media Laws 3, published online on October 14.

Cohen, Stanley (1972) Folk Devils and Moral Panics. London: Routledge.

Colbert Report (2005) ‘The WØRD “Truthiness”, Comedy Central, 17 October.

 179 Collins, Ben, and John Russell (2018) ‘Russians Used Reddit and Tumblr to Troll the 2016 Election’,

The Daily Beast, 2 March. https://www.thedailybeast.com/russians-used-reddit-and-tumblr-totroll-the-2016-election.

Collins, Keith Sheera Frenkel (2018) ‘Can you spot the deceptive Facebook post?’ New York Times, 4 September.

Confessore, Nicholas and Daisuke Wakabayashi (2017)’How Russia Harvested American Rage to

Reshape U.S. Politics’, New York Times, 9 October. Covington, Paul, Jay Adams, and Emre Sargin (2016) ‘Deep Neural Networks for YouTube

Recommendations’, Proceedings of the 10th ACM Conference on Recommender Systems (RecSys '16) New York, NY: ACM, pp.191-198. DOI: https://doi.org/10.1145/2959100.2959190

Dahlgren, Kristen and Daniel Arkin (2017) ‘11-Year-Old Texas Boy Invents Device to Prevent Hot Car Deaths’, NBC News, 29 June.

Dan, Ovidiu and Brian D. Davison (2016) ‘Measuring and Predicting Search Engine Users’

Satisfaction’, ACM Computing Surveys, 49(1), art. 18. Daniels, Jessie (2018) ‘The algorithmic rise of the alt-right’, Contexts, 17(1): 60-65. Darnton, Robert (2010) Poetry and the Police: Communication Networks in Eighteenth-Century Paris.

Cambridge, MA: Harvard University Press. _____ (2017) ‘The True History of Fake News’, New York Review of Books, 13 February. Day, Amber and Ethan Thompson (2012) ‘Live from New York, it’s the fake news! Saturday night

live and the (non)politics of parody,’ Popular Communication 10(1-2): 170-182. DeAmicis, Carmel (2014) ‘Facebook shifts its algorithm to fight clickbait. Will it kill off Upworthy

and Buzzfeed?’, GIGAOM, Austin: Giga Omni Media. De Hoax-Wijzer (n.d.) ‘Valse Nieuwssites’. https://sites.google.com/site/dehoaxwijzer/valsenieuwssites.

Digital Methods Initiative (2015) ‘Instagram scraper’.

https://wiki.digitalmethods.net/Dmi/ToolInstagramScraper DiResta, Renee, Kris Shaffer, Becky Ruppel, et al. (2018) ‘The Tactics & Tropes of the Internet

Research Agency’, Report, New Knowledge. https://disinformationreport.blob.core.windows.net/disinformation-report/NewKnowledge Disinformation-Report-Whitepaper.pdf.

Drewe, Nick (2016) ‘The Hilarious List Of Hashtags Instagram Won’t Let You Search’, The Data

Pack, 10 May. http://thedatapack.com/banned-instagram-hashtags-update/ Drog (2018) ‘Slecht Nieuws: Serious game over propaganda’, Website,

https://www.slechtnieuws.nl/. Duke Reporters Lab (2019) ‘Fact Checking News’, Website. https://reporterslab.org/factchecking/

  Dumont, P. and Kies, R. (2012) ‘Smartvote.lu: Usage and impact of the first VAA in Luxembourg’,

International Journal of Electronic Governance 5(3/4): 388–410. Dupuy, Lisa (2018) ‘GeenStijl Staat Niet Meer Op EU-Lijst Nepnieuws’, NRC Handelsblad, 29

January. https://www.nrc.nl/nieuws/2018/01/29/geenstijl-staat-niet-meer-op-eu-lijstnepnieuws-a1590216.

Edelman (2018) ‘2018 Edelman Trust Barometer’, Report. https://www.edelman.com/sites/g/files/aatuss191/files/2018- 10/Edelman_Trust_Barometer_Employee_Experience_2018_0.pdf

Ellick, Adam B. and Adam Westbrook (2018) ‘Operation Infektion’, New York Times, 12 November.

 180 EU HLEG (2018) A multi-dimensional approach to disinformation, Report of the independent High level

Group on fake news and online disinformation, Luxembourg: Publications Office of the European Union.

EU vs Disinfo. 2017. ‘Sputnik’s Short-Lived Presence in the Slovak Press Agency’. 2017. EU vs

Disinfo (blog). 1 April. https://euvsdisinfo.eu/sputniks-short-lived-presence-in-the-slovak-pressagency/.

Fandos, Nicholas (2017) ‘White House Pushes ‘Alternative Facts.’ Here Are the Real Ones’, New

York Times, 22 January. Faris, Robert M., Hal Roberts, Bruce Etling, Nikki Bourassa, Ethan Zuckerman, and Yochai Benkler

(2017) ‘Partisanship, Propaganda, and Disinformation: Online Media and the 2016 U.S. Presidential Election’, Berkman Klein Center for Internet & Society Research Paper.

Farkas, Johan and Marco Bastos (2018) ‘IRA Propaganda on Twitter: Stoking Antagonism and

Tweeting Local News’, SMSociety ‘18, Copenhagen. Feldman, Brian (2017) ‘In Russia, You Can Buy Instagram Likes From a Vending Machine’, New

York Magazine, 8 June. Feuz, Martin, Matthew Fuller, and Felix Stalder (2011) ‘Personal Web Searching in the Age of

Semantic Capitalism: Diagnosing the Mechanisms of Personalisation’, First Monday, 16(2), 7 February. https://doi.org/10.5210/fm.v16i2.3344.

Fink, Christina (2018) ‘Dangerous Speech, Anti-Muslim Violence, and Facebook in Myanmar’,

Journal of International Affairs 71(1.5): 43–52. Finkel, Jacob, Steven Jiang, Luo Mufan, Rebecca Mears, Danaë Metaxa-Kakavouli, Camille Peeples,

Brendan Sasso, Arjun Shenoy, Vincent Sheu, and Nicolás Torres-Echeverry (2017) Fake News

and Misinformation: The Roles of the Nation’s Digital Newsstands, Facebook, Google, Twitter and Reddit,

Stanford, CA: Stanford Law School. https://www-cdn.law.stanford.edu/wpcontent/uploads/2017/10/Fake-News-Misinformation-FINAL-PDF.pdf

FireEye (2018) ‘Suspected Iranian Influence Operation’, Report, Milpitas, CA: FireEye.

Fitts, Alexis Sobel (2017) ‘Welcome to the Wikipedia of the Alt-right’, Wired, 21 June.

Fletcher, Richard, Alessio Cornia, Lucas Graves, and Rasmus Kleis Nielsen (2018) ‘Measuring the

reach of “fake news” and online disinformation in Europe’, Reuters Institute for the Study of

Journalism, Oxford University, February.

Flynn, Kerry (2018) ‘Facebook will elevate “trusted” news outlets after surveying U.S. users’,

Mashable, 19 January.

Foucault, Michel (2008) The Birth of Biopolitics: Lectures at the Collège De France, 1978-1979, New York,

NY: Palgrave Macmillan.

Frenkel, Sheera, Kate Conger and Kevin Roose (2019) ‘Russia’s Playbook for Social Media

Disinformation Has Gone Global’, New York Times, 31 January.

Frenkel, Sheera, Nicholas Casey and Paul Mozur (2018) ‘In Some Countries, Facebook’s Fiddling

Has Magnified Fake News’, New York Times, 14 January.

Fubini, Federico (2018) ‘Tweet populisti dalla Russia sulla politica italiana. Come negli Usa’, Corriere

della Sera, 1 August.

Gartenberg, Chaim (2018) ‘Facebook is going to start surveying users to determine trustworthy news

sources’, The Verge, 19 January.

Garzia, Diego and Marschall, Stefan (2012) ‘Voting Advice Applications Under Review: The State of

Research’, International Journal of Electronic Governance, 5: 203-22.

 181 German Federal Foreign Office (2018) ‘Fake news, bots and provocative statements –

disinformation on the internet’, Auswaertiges Amt website, published online on August 7. Gessen, Masha (2018) ‘Why the Russian Influence Campaign Remains So Hard to Understand’, The

New Yorker, 18 December. Gitelman, Lisa (2006) Always Already New: Media, History, and the Data of Culture. Cambridge, MA:

MIT Press. Glowacki, Monika, Vidya Narayanan, Sam Maynard, Gustavo Hirsch, Bence Kollanyi, Lisa-Maria

Neudert, Phil Howard, Thomas Lederer, and Vlad Barash (2018) ‘News and Political Information Consumption in Mexico: Mapping the 2018 Mexical Presidential Election on Twitter and Facebook,’ 2018.2, COMPROP Data Memo. Oxford: Oxford Internet Institute.

Goodwin, Matthew and Roger Eatwell (2018) National Populism: The Revolt Against Liberal Democracy, London: Penguin Books.

Gottfried, Jeffrey and Elisa Shearer (2016) ‘News Use Across Social Media Platforms 2016’,

Washington, DC: Pew Research Center. Government Offices of Sweden (2017) A practical approach on how to cope with disinformation,

Government Offices of Sweden website, published online on October 6. Gramer, Robbie (2017) ‘Denmark Creates the World’s First Ever Digital Ambassador’, Foreign Policy,

published online on January 27. Graves, Lucas (2016) Deciding What’s True: The Rise of Political Fact-Checking in American Journalism. New

York: Columbia University Press. Graves, Lucas and Federica Cherubini (2016) The Rise of Fact-checking Sites in Europe. Oxford:

University of Oxford, Reuters Institute for the Study of Journalism. Grinberg, Nir, Kenneth Joseph, Lisa Friedland, Briony Swire-Thompson and David Lazer (2019)

‘Fake news on Twitter during the 2016 U.S. presidential election’, Science 363(6425): 374-378. Groll, Elias (2018) ‘Battling the Bots’, Foreign Policy, 12 November.

Groot, Tim, Sophie Minihold, Jessica Robinson, Manuel Schneider, Joanna Sleigh and Dydimus

Zengenene (2019). ‘Russia, Twitter & Authenticity: Establishing Credibility Metrics’, Digital Methods Initiative, Winter School 2019. https://wiki.digitalmethods.net/Dmi/WinterSchool2019CredibilityMetrics

Guess, Andrew, Brendan Nyhan and Jason Reifler (2018) ‘Selective Exposure to Misinformation:

Evidence from the consumption of fake news during the 2016 U.S. presidential campaign’,

Report, Hannover, NH: Dartmouth College, https://www.dartmouth.edu/~nyhan/fake-news

2016.pdf.

Haciyakupoglu, Gulizar, Jennifer Yang Hui, V. S. Suguna, Dymples Leong, and Muhammad Faizal

Bin Abdul Rahman (2018) Countering Fake News: A Survey Of Recent Global Initiatives, Singapore:

Nanyang Technological University.

Hagen, Sal, Daniël de Zeeuw, Stijn Peeters, Emilija Jokubauskaitė, and Ángeles Briones (2019).

‘Understanding Normiefication: A Cross-Platform Analysis of the QAnon Conspiracy Theory’,

Digital Methods Initiative, 21 February.

https://wiki.digitalmethods.net/Dmi/WinterSchool2019Normiefication.

Haidt, Jonathan (2012) The Righteous Mind: Why Good People Are Divided by Politics and Religion. New

York: Pantheon Books.

Haim, Mario, Andreas Graefe and Hans-Bernd Brosius (2018) ‘Burst of the Filter Bubble? Effects of

personalization on the diversity of Google News’, Digital Journalism, 6(3):1-14.

 182 Halavais, Alexander M. Campbell (2017) Search Engine Society, Second edition, Cambridge, UK: Polity

Press. Hall, Jim (2001) Online journalism: a critical primer. London: Pluto Press. Hameleers, Michael and Toni van der Meer (2019) ‘Fact-checks helpen tegen desinformatie! Maar

dan moeten ze wel gezien worden’, De Nieuwe Reporter, 29 January. Hankes, Keegan, and Alex Amend (2018) ‘The Alt-Right Is Killing People’, Southern Poverty Law

Center, 5 February. https://www.splcenter.org/20180205/alt-right-killing-people. Haque, Md Mahfuzul, Mohammad Yousuf, Zahedur Arman, Md Main Uddin Rony, Ahmed Shatil

Alam, Kazi Mehedi Hasan, Md Khadimul Islam, and Naeemul Hassan (2018) ‘Fact-Checking Initiatives in Bangladesh, India, and Nepal: A Study of User Engagement and Challenges’, ArXiv:1811.01806 [Cs], November. http://arxiv.org/abs/1811.01806.

Harsin, Jayson (2018) ‘A Critical Guide to Fake News: From Comedy to Tragedy’, Pouvoirs,

2018/1:164, 99-119.

Harvey, Del and Yoel Roth (2018) ‘An update on our elections integrity work’, Twitter blog, 1

October. Hatewatch Staff (2015) ‘Getting Cucky: A Brief Primer On The Radical Right's Newest

'cuckservative' Meme’, Southern Poverty Law Center, 7 August. https://www.splcenter.org/hatewatch/2015/08/07/getting-cucky-brief-primer-radical-rightsnewest-cuckservative-meme

Hawley, George (2017) Making Sense of the Alt-Right, New York: Colombia University Press.

Haynes, Gavin (2016) ‘Can you spot the ‘real’ fake news story?’ The Guardian, 28 December.

Heck, Wilmer (2018) ‘Nederlandse media dagen EU voor rechter na beschuldigingen desinformatie’, NRC Handelsblad, 20 February.

Heck, Wilmer (2017) ‘Rusland Beïnvloedt Ons Vooral Online’, NRC Handelsblad, 8 January.

Hedman, Freja, Fabian Sivnert, Lisa-Maria Neudert, Bence Kollanyi, Philip N. Howard and Vidya

Narayanan (2018) ‘News and Political Information Consumption in Sweden: Mapping the 2018 Swedish General Election on Twitter’, Computational Propaganda Data Memo, Oxford: Oxford Internet Institute, 6 September.

Hedrick, Ashley, Dave Karpf and Daniel Kreiss (2018) ‘The Earnest Internet vs. the Ambivalent

Internet’, International Journal of Communication, 12:1057-1064. Heikkilä, Niko (2017) ‘Online Antagonism of the Alt-Right in the 2016 Election,’ European Journal of

American Studies 12(2): 1-23, DOI:10.4000/ejas.12140. Helmond, Anne (2015) ‘The Platformization of the Web: Making Web Data Platform Ready’, Social

Media + Society 1(2):1-11. https://doi.org/10.1177/2056305115603080. Hemmer, Nicole (2016) Messengers of the Right: Conservative Media and the Transformation of American

Politics, Philadelphia: University of Pennsylvania Press. Hendricks, Vincent F. and Mads Vestergaard (2019) Reality Lost: Markets of Attention, Misinformation

and Manipulation, Cham: Springer. Herderscheê, Gijs, and Remco Meijer (2019) ‘Kabinet krijgt bitter voorproefje van regeren zonder

meerderheid in Eerste Kamer’, De Volkskrant. January 31. https://www.volkskrant.nl/gsb378d420.

Hern, Alex (2017) ‘Google acts against fake news on search engine’, The Guardian, 25 April.

 183 Hern, Alex and Jim Waterson (2018) ‘Facebook cracks down on ‘dark ads’ by British political

groups’, The Guardian, 16 October.

Herrman, John (2016) ‘Inside Facebook’s (Totally Insane, Unintentionally Gigantic, Hyperpartisan) Political-Media Machine’, The New York Times, 28 August.

https://www.nytimes.com/2016/08/28/magazine/inside-facebooks-totally-insaneunintentionally-gigantic-hyperpartisan-political-media-machine.html.

Hindman, Matthew Scott (2008) The Myth of Digital Democracy, Princeton: Princeton University Press. Hirst, Martin (2017) ‘Towards a political economy of fake news’, The Political Economy of Communication

5(2), 82–94. Hirzalla, Fadi and Liesbet van Zoonen (2015) ‘Online voting applications. Foci, findings and future

of an emerging research field’ in Coleman, S. & Freelon, D. (eds), Handbook of Digital Politics, Northampton, MA: Edward Elgar: 87-103.

Holt, Jared (2017) ‘White Supremacy Figured Out How to Become YouTube Famous | Right Wing Watch,’ October. http://www.rightwingwatch.org/report/white-supremacy-figured-out-how-tobecome-youtube-famous/.

Holzman, Michael (2008) James Jesus Angleton, the CIA and the Craft of Counterintelligence, Amherst:

University of Massachusetts Press. Honeycutt, Courtenay and Susan C. Herring (2009) ‘Beyond microblogging. Conversation and

collaboration’, 42nd Hawaii International Conference on System Sciences. Los Alamitos, CA: IEEE Press.

House of Commons (2019) ‘Disinformation and “Fake News”: Final Report Eighth Report of

Session 2017–19’, Digital, Culture, Media and Sport Committee, London: UK Parliament. Howard, Philip N., Gillian Bolsover, Bence Kollanyi, Samantha Bradshaw, and Lisa-Maria Neudert

(2017) ‘Junk news and bots during the U.S. election: What were Michigan voters sharing over Twitter?’ Computational Propaganda Data Memo, Oxford: Oxford Internet Institute.

Howard, Philip N., Bharath Ganesh, Dimitra Liotsiou, John Kelly and Camille François (2018) ‘The IRA, Social Media and Political Polarization in the United States, 2012-2018’, Report,

Computational Propaganda Research Project, Oxford: Oxford Internet Institute. Hughes, Taylor, Jeff Smith, and Alex Leavitt (2018) ‘Helping People Better Assess the Stories They

See in News Feed with the Context Button | Facebook Newsroom’, Facebook Newsroom (blog). April. https://newsroom.fb.com/news/2018/04/news-feed-fyi-more-context/

Instaloader (2019). Instaloader software, version 4.2.5, GitHub project, https://instaloader.github.io/

Isaac, Mike (2016) ‘Facebook, in Cross Hairs After Election, Is Said to Question Its Influence’, New York Times, 12 November.

Jack, Caroline (2017) Lexicon of Lies: Terms for Problematic Information New York: Data & Society

Research Institute.

Jamieson, Kathleen Hall. 2018. Cyberwar: How Russian Hackers and Trolls Helped Elect a

President. Oxford: Oxford University Press.

Jankowski, Nicholas W. (2018) ‘Researching Fake News: A Selective Examination of Empirical

Studies’, Javnost - The Public, 25(1-2): 248-255.

 184 Jansen, Bernard J. and Amanda Spink (2003) ‘An Analysis of Web Documents Retrieved and

Viewed’, 4th International Conference on Internet Computing, Las Vegas, Nevada, 23-26 June, pp. 65-69.

Java, Akshay, Xiaodan Song, Tim Finin and Belle Tseng (2007) ‘Why we twitter: understanding microblogging usage and communities’. In Proceedings of the 9th WebKDD and 1st SNA-KDD 2007 workshop on Web mining and social network analysis, New York, NY: ACM, pp. 56-65.

Jeangène Vilmer, Jean-Baptiste, Alexandre Escorcia, Marine Guillaume, Janaina Herrera (2018)

Information Manipulation: A Challenge for Our Democracies, report by the Policy Planning Staff (CAPS) of the Ministry for Europe and Foreign Affairs and the Institute for Strategic Research (IRSEM) of the Ministry for the Armed Forces, Paris, August.

Johnson, Brett G., and Kimberly Kelling (2018) ‘Placing Facebook’ Journalism Practice 12(7): 817–33. https://doi.org/10.1080/17512786.2017.1349546

Jonker, Jorn (2017) ‘Nepnieuws gevaar voor Nederland’, De Telegraaf, 14 November.

Kaiser, Jonas, and Adrian Rauchfleisch (2018) ‘Unite the Right? How YouTube’s Recommendation Algorithm Connects the U.S. Far-Right.’ Data & Society Media Manipulation, April 11.

doi:10.1093/acprof:oso/9780199641260.001.0001/acprof-9780199641260. Kanne, Peter and Milan Driessen (2017) Desinformatie leidt tot verwarring bij nieuwsconsument, Amsterdam:

I&O Research. Kasteleijn, Nando (2017) ‘Nepnieuws groot gevaar voor Nederland? Dat lijkt dus wel mee te vallen’,

NOS, 28 August. van Keulen, Ira, Iris Korthagen, Paul Diederen en Pieter van Boheemen (2018) ‘Digitalisering van

het nieuws: Online nieuwsgedrag, desinformatie en personalisatie in Nederland’, Den Haag: Rathenau Instituut.

King, Gary and Nathaniel Persily (2018) ‘A New Model for Industry-Academic Partnerships’,

Working Paper, 9 April. http://j.mp/2q1IQpH Kist, Reinier (2019) ‘Factchecken Facebook loopt stuk op aansprakelijkheid’, NRC Handelsblad, 26

February. Kist, Reinier and Rik Wassens (2017) ‘Ook Nederlandse media werden misleid door Russische

trollen’, NRC Handelsblad, 8 December. Kist, Reinier and Rik Wassens (2018a) ‘Russische trollen actief in Nederland’, NRC Handelsblad, 15

July. Kist, Reinier and Rik Wassens (2018b) ‘Russisch trollenleger ook actief in Nederland’, NRC

Handelsblad, 15 July. Kist, Reinier and Peter Zantingh (2017) ‘Geen grote rol nepnieuws in aanloop naar verkiezingen’,

NRC Handelsblad, 6 March. https://www.nrc.nl/nieuws/2017/03/06/fake-news-nee-zo-erg-ishet-hier-niet-7144615-a1549050

Kitta, Andrea (2018) ‘Alternative Websites and Fake News: Taking a Stab at Definition, Genre and Belief’, The Journal of American Folklore 131(522): 405-412.

Komok, Anna (2018) ‘How to Check Instagram Account for Fake Followers’, HypeAuditor, 6 July. https://hypeauditor.com/blog/how-to-check-instagram-account-for-fake-followers/

Kouwenhoven, Andreas and Hugo Logtenberg (2017) ‘Hoe Denk met ‘trollen’ politieke tegenstanders monddood probeert te maken’, NRC Handelsblad, 10 February.

Kranenberg, Annieke (2017) Wie weet nog wat er waar is?, De Volkskrant, 23 December.

 185 Kranenberg, Annieke, and Hassan Bahara (2018) ‘Hoe alt-right online Jodenhaat verspreidt’, De

Volkskrant, 9 November. https://www.volkskrant.nl/gs-b1714693 Kwak, Haewoon, Changhyun Lee, Hosung Park and Sue Moon (2010). ‘What is Twitter? A social

network or a news media?’, in Proceedings of the 19th International Conference on World Wide Web, New York: ACM, April, pp. 591-600.

Lagorio-Chafkin, Christine (2018) ‘Reddit Confirms New Russian Meddling Efforts’, Inc., 4 October. https://www.inc.com/christine-lagorio/reddit-finds-new-russian-interference-campaign.html.

Laquintano, Timothy and Annette Vee (2017) ‘How Automated Writing Systems Affect the

Circulation of Political Information Online’, Literacy in Composition Studies, 5(2): 43-62. Latour, Bruno (2008) What is the style of matters of concern? Assen: Van Gorcum.

Latour, Bruno, and Steve Woolgar (1979) Laboratory Life: The Social Construction of Scientific Facts.

Princeton, NJ: Princeton University Press. Lazarsfeld, Paul F., Bernard R. Berelson and Hazel Gaudet (1948) The People’s Choice: How the Voter

Makes Up His Mind in a Presidential Campaign, New York, NY: Columbia University Press. Lazer, David M.J., Matthew A. Baum, Yochai Benkler, Adam J. Berinsky, Kelly M. Greenhill, Filippo

Menczer, Miriam J. Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, Michael Schudson, Steven A. Sloman, Cass R. Sunstein, Emily A. Thorson, Duncan J. Watts and Jonathan L. Zittrain (2018) ‘The science of fake news’, Science, 359(6380):1094-1096.

Lee, Alexis (2019) ‘What Is a ‘Facebook Engagement’?’, BuzzSumo Knowledge Base. https://help.buzzsumo.com/faqs-and-troubleshooting/product-faqs/what-is-a-facebookengagement.

Lewis, Rebecca (2018) ‘Alternative Influence: Broadcasting the Reactionary Right on YouTube’, Data & Society. https://datasociety.net/wp-content/uploads/2018/09/DS_Alternative_Influence.pdf.

Liotsiou, Dimitra, Bence Kollanyi, and Philip N. Howard (2019) ‘The Junk News Aggregator:

Examining Junk News Posted on Facebook, Starting with the 2018 US Midterm Elections’, ArXiv:1901.07920 [Cs], January. http://arxiv.org/abs/1901.07920.

Lippmann, Walter (1922) Public Opinion, New York, NY: Harcourt, Brace and Co.

Lomas, Natasha (2018) ‘Google is surfacing Texas shooter misinformation in search results — thanks also to Twitter’, TechCrunch, 11 June.

Lorenz, Taylor (2014) ‘Instagram Rapture Claims Millions Of Celebrity Instagram Followers’,

Business Insider, 18 December. Lorenz, Taylor (2018) ‘Instagram’s Christmas Crackdown. No meme account is safe—not even

@God’, The Atlantic, 27 December. Louwerse, Tom and Martin Rosema (2014) ‘The design effects of voting applications: Comparing

methods of calculating matches’, Acta Politica 49(3):286-312. Lüders, Marika, Lin Prøitz and Terje Rasmussen (2010) ‘Emerging personal media genres’, New Media & Society, 12(6): 947–963.

Lynskey, Dorian (2018) ‘How Dangerous Is Jordan B Peterson, the Rightwing Professor Who 'Hit a Hornets' Nest'?’ The Guardian, February 7.

https://www.theguardian.com/science/2018/feb/07/how-dangerous-is-jordan-b-peterson-therightwing-professor-who-hit-a-hornets-nest.

MacFarquhar, Neil (2018) ‘A Powerful Russian Weapon: The Spread of False Stories’, The New York Times, 20 January. https://www.nytimes.com/2016/08/29/world/europe/russia-swedendisinformation.html.

 186 Maheshwari, Sapna (2018) ‘Uncovering Instagram Bots With a New Kind of Detective Work’,

The New York Times, 12 March.

Malice, Michael (2019) The New Right: A Journey to the Fringe of American Politics, New York, NY: All

Points Books. Marchal, Nahema, Lisa-Maria Neudert, Bence Kollanyi, and Philip N. Howard (2018) ‘Polarization,

Partisanship and Junk News Consumption on Social Media During the 2018 US Midterm Elections’ 2018.5, COMPROP Data Memo, Oxford: Oxford Internet Institute.

Margolis, Joseph (1995) ‘Beyond postmodernism: Logic as rhetoric’, Argumentation, 9(1): 21-31.

Marres, Noortje (2018) ‘Why We Can’t Have Our Facts Back’, Engaging Science, Technology, and Society, 4: 423-443.

Marres, Noortje, and Esther Weltevrede (2013), ‘Scraping the Social?’ Journal of Cultural Economy, 6(3): 313–35, DOI:10.1080/17530350.2013.772070.

Martens, Bertin, Luis Aguiar, Estrella Gomez-Herrera and Frank Mueller-Langer (2018) ‘The Digital Transformation of News Media and the Rise of Disinformation and Fake News,’ SSRN Electronic Journal, DOI:10.2139/ssrn.3164170.

Marvin, Carolyn (1988) When Old Technologies were New. New York, NY: Oxford University Press.

Marwick, Alice E. (2018) ‘Why Do People Share Fake News? A Sociotechnical Model of Media

Effects’, Georgetown Law Technology Review, 2(2): 474-512. Marwick, Alice, and Rebecca Lewis (2017) ‘Media Manipulation and Disinformation Online’, Data &

Society, New York. Matsakis, Louise (2017) 'This Russian Vending Machine Will Sell You Fake Instagram Likes',

Motherboard Vice, 7 June. Mauri, Michele, Tommaso Elli, Giorgio Caviglia, Giorgio Uboldi, and Matteo Azzi (2017)

‘RAWGraphs: A Visualisation Platform to Create Open Outputs’, in Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter, 28:1–28:5. CHItaly ’17, New York, NY: ACM. https://doi.org/10.1145/3125571.3125585.

McGarry, Caitlin (2013) ‘Twitter bots, fake retweets rake in big bucks’, PCWorld, 10 April.

McGonagle, Tarlach (2017) ‘“Fake news”: False fears or real concerns?’, Netherlands Quarterly of

Human Rights, 35(4): 203–209. McKernon, Edward (1925) ‘Fake News and the Public’, The Harper’s Monthly, October, pp. 528-536. McKernon, Edward (1928) ‘News Fakers’, The Outlook, 149(4): 130-141. McNeil, Maureen (2013) ‘Between a Rock and a Hard Place: The Deficit Model, the Diffusion

Model and Publics in STS’, Science as Culture, 22(4): 589-608. McQueen, Sharon (2018) ‘From Yellow Journalism to Tabloids to Clickbait: The Origins of Fake

News in the United States’, in Agosto, Denise E. (ed.) Information Literacies and Libraries in the Age of Fake News, Santa Barbara, CA: Libraries Unlimited, pp. 12-35.

Merrill, Jeremy B. and Ariana Tobin (2019) ‘Facebook Moves to Block Ad Transparency Tools —

Including Ours’, ProPublica, 29 January. Milneil, Christian (2018) ‘Data: Read the tweets from alleged Russian troll accounts targeting Maine

politicians’, Portland Press Herald, 2 August. Mina, An Xiao (2019). Memes to Movements: How the World’s Most Viral Media Is Changing Social Protest

and Power, Boston, MA: Beacon Press. Ministerie van Binnenlandse Zaken en Koninkrijksrelaties (2019) ‘Campagne nepnieuws vandaag van

start - Nieuwsbericht’, Rijksoverheid.nl. March 11.

 187 https://www.rijksoverheid.nl/actueel/nieuws/2019/03/11/campagne-nepnieuws-vandaag-vanstart.

Misérus, Mark and Robert van der Noordaa (2018a) ‘Het trollenleger van popartiest Dotan’, De

Volkskrant, 14 April. Misérus, Mark and Robert van der Noordaa (2018b) ‘Dotan erkent aanmaken nepfans: “Ik was heel

naïef, veel te ambitieus en onzeker”’, De Volkskrant, 16 April. Monbiot, George (2019) ‘Dark money is pushing for a no-deal Brexit. Who is behind it?’, The

Guardian, 13 February. Morozov, Evgeny (2017) ‘Moral panic over fake news hides the real enemy – the digital giants’, The

Guardian, 8 January. Mosseri, Adam (2017a) ‘Working to Stop Misinformation and False News | Facebook Newsroom’,

Facebook Newsroom, blog post, April. https://newsroom.fb.com/news/2017/04/working-to-stopmisinformation-and-false-news/.

———. 2017b. “Showing More Informative Links in News Feed | Facebook Newsroom.” Facebook Newsroom, blog post, June 30. https://newsroom.fb.com/news/2017/06/news-feed-fyi-showingmore-informative-links-in-news-feed/

Mudde, Cas, and Cristóbal Rovira Kaltwasser (2017) Populism: a Very Short Introduction, Oxford:

Oxford University Press.

Muirhead, Russell, and Nancy L. Rosenblum (2019) A Lot of People Are Saying: The New Conspiracism and the Assault on Democracy, Princeton, NJ: Princeton University Press.

Muller, Jan-Werner (2016) What Is Populism? Philadelphia, PA: University of Pennsylvania Press.

Nagle, Angela (2017) Kill All Normies: Online Culture Wars From 4Chan and Tumblr to Trump and the Alt Right. London: Zero Books.

Neudert, Lisa-Maria N. (2017) ‘Computational Propaganda in Germany: A Cautionary Tale’,

Computational Propaganda Working Paper, 2017.7, COMPROP Data Memo, Oxford: Oxford Internet Institute.

Neudert, Lisa-Maria, Bence Kollanyi, and Philip N. Howard (2017) ‘Junk News and Bots during the German Parliamentary Election: What Are German Voters Sharing over Twitter?’, 2017.7,

COMPROP Data Memo, Oxford: Oxford Internet Institute.

New Knowledge (2018) ‘The Tactics & Tropes of the Internet Research Agency’, White Paper,

Austin, TX: New Knowledge.

Newman, Michelle C. (2010) ‘The Daily Show and Meta-Coverage: How Mock News Covers the

Political Communications System’, The Elon Journal of Undergraduate Research in Communications, 1(2).

NewsGuard (2019), ‘Restoring Trust and Accountability’, webpage, https://www.newsguardtech.com.

Niederer, Sabine (2017) ‘The Study of Networked Content: Five Considerations for Digital Research in the Humanities,’ in Giovanni Schiuma and Daniela Carlucci (eds.), Big Data in the Arts and

Humanities: Theory and Practice, Boca Raton, FL: CRC Press, pp. 89-100. Nijmeijer, Bert (2018) ‘Zelf nepnieuws maken om daarna de echte trollen te kunnen herkennen’,

NRC Handelsblad, 22 July.

 188 Noble, Safiya Umoja (2018) Algorithms of Oppression: How Search Engines Reinforce Racism. New York,

NY: New York University Press. van der Noorda, Robert van der en Coen van de Ven (2019) ‘Het MH17 Complot’, De Groene

Amsterdammer, 29 May. van der Noorda, Robert and Coen van de Ven (2018a) ‘Hoe Russische trollen inspelen op westerse

angsten’, De Groene Amsterdammer, 28 Augustus. van der Noordaa, Robert and Coen van de Ven (2018b) ‘3 Miljoen “trollentweets” onderzocht: hoe

Rusland via sociale media ook in België verdeeldheid zaait’, Knack, 27 November. NOS (2017a) ‘Baby van 20 kilo heeft de primeur: eerste nepnieuws in Nederland’, NOS, 8 June. NOS (2017b) ‘Ollongren: Russische desinformatie bij Oekraïne-referendum’, NOS, 15 November. NOS (2018) ‘Twitters grote schoonmaak: Wilders en Denk-politici verliezen volgers’, NOS, 13 July. Nuzzi, Olivia (2017) ‘Kellyanne Conway Is a Star’, New York Magazine, March.

Nyhan, Brendan, Ethan Porter, Jason Reifler and Thomas J. Wood (2019) ‘Taking Fact-Checks

Literally But Not Seriously? The Effects of Journalistic Fact-Checking on Factual Beliefs and Candidate Favorability’, Political Behavior, published online 21 January.

O’Brien, Chris (2019) ‘Sheryl Sandberg says Facebook is now blocking 1 million fake accounts every day’, Venture Beat, 21 January.

O’Donovan, Caroline (2014) ‘What is clickbait?’, Niemanlab, 25 August.

Omnicore. (2019) ‘Twitter by the numbers’, Omnicore

Agency. https://www.omnicoreagency.com/twitter-statistics/ Opper, F. (1894) ‘The fin de siècle newspaper proprietor’, Puck, 35(887). Oudenampsen, Merijn (2013) ‘Met de Tjoeki Tjoeki Naar Takki Takki’. De Groene Amsterdammer, 3

July. https://www.groene.nl/artikel/met-de-tjoeki-tjoeki-naar-takki-takki. Panagopoulos, Costas (2016) ‘All about that base: Changing campaign strategies in US presidential

elections’ Party Politics, 22(2): 179–90. Pariser, Eli (2011) The Filter Bubble: What the Internet Is Hiding From You. New York, NY: Penguin. Parlapiano, Alicia and Jasmine C. Lee (2018) ‘The Propaganda Tools Used by Russians to Influence

the 2016 Election’, New York Times, 16 February. Peck, Reece (2019) Fox Populism: Branding Conservatism as Working Class, Cambridge: Cambridge

University Press. Pedersen, David B. and Vincent F. Hendricks (2014) ‘Science Bubbles’, Philosophy and Technology,

27(4): 503-518. Peek, Simone (2018) ‘NOS-Hoofdredacteur Over Bericht Soros: “Zo Had Het Niet Gemoeten”’,

NRC Handelsblad, 24 October. Peel, Michael (2019) ‘EU election suffered Russian disinformation, Brussels finds’, Financial Times, 14

June. Persily, Nathaniel (2017) ‘The 2016 U.S. Election: Can Democracy Survive the Internet?’. Journal of

Democracy, 28(2): 63-76 Phillips, Whitney (2015) This Is Why We Can’t Have Nice Things: Mapping the Relationship Between Online

Trolling and Mainstream Culture. Cambridge, MA: MIT Press. Phillips, Whitney (2018) ‘The Oxygen of Amplification: Better Practices for Reporting on

Extremists, Antagonists and Manipulators Online’, Report, New York: Data & Society Research Institute.

Phillips, Whitney and Ryan Milner (2018) The Ambivalent Internet: Mischief, Oddity, and Antagonism

Online, Cambridge: Polity Press.

 189 Pleijter, Alexander (2017) ‘De nepnieuwslawine zonder nepnieuws’, Villamedia, 17 November.

Pohjonen, Matti and Sahana Udupa (2017) ‘Extreme Speech Online: An Anthropological Critique of Hate Speech Debates’, International Journal of Communication, 11: 1173-1191.

Posetti, Julie and Alice Matthews (2018) A short guide to the history of ‘fake news’ and disinformation. A learning module for journalists and journalism educator, Washington, DC: International Center for

Journalists. Poynter (2019) A guide to anti-misinformation actions around the world, St. Petersburg, FL: The Poynter

Institute. Puschmann, Cornelius (2018) ‘Beyond the Bubble: Assessing the Diversity of Political Search

Results’ Digital Journalism, published online 28 November. https://doi.org/10.1080/21670811.2018.1539626

Puschmann, Cornelius (2017) ‘How significant is algorithmic personalization in searches for political parties and candidates?’, Algorithmed Public Spheres blog, https://aps.hans-bredowinstitut.de/personalization-google/

  Ratkiewicz, Jacob, Michael D. Conover, Mark Meiss, Bruno Goncalves, Alessandro Flammini,

Filippo Menczer Menczer (2011) ‘Detecting and Tracking Political Abuse in Social Media’, Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media, Barcelona, Spain, July.

Reid, John, Bastiaan Geleijnse and Jean-Marc van Tol (2018) ‘Fokke en Sukke hebben zitting in de taskforce nepnieuws’, Cartoon, 17 January.

Renner, Nausicaa (2017) ‘Memes trump articles on Breitbart’s Facebook page’, Columbia Journalism

Review, 30 January. Rieder, Bernhard (2018) “Facebook’s app review and how independent research just got a lot

harder”. Politics of Systems blog, 11 August. _____ (2012). The refraction chamber: Twitter as sphere and network. First Monday, 17(11). _____, Ariadna Matamoros-Fernández and Òscar Coromina (2018). “From ranking algorithms to

‘ranking cultures’: Investigating the modulation of visibility in YouTube search results.” Convergence: The International Journal of Research into New Media Technologies, 24(1): 50–68.

_____, Rasha Abdulla, Thomas Poell, Robbert Woltering, Liesbeth Zack (2016) ‘Data Critique and Analytical Opportunities for Very Large Facebook Pages. Lessons Learned from Exploring “We Are All Khaled Said”’, Big Data & Society, 2.2: 1-22

RNTC (2019). ‘Challenging fake news’, webpage, Hilversum: RNTC.

Https://rntc.com/blog/challenging-fake-news. Roberts, Sarah T. (2016) ‘Commercial Content Moderation: Digital Laborers’ Dirty Work’, in Safiya

Umoja Noble and Brendesha M. Tynes (eds.) The Intersectional Internet, New York, NY: Peter Lang, pp. 147-160.

Robertson, Adi (2017) ‘After its 4chan slip-up, is it time for Google to drop Top Stories?’, The Verge, 3 October.

RoBhat Labs (2017) ‘Identifying Propaganda Bots on Twitter’, Medium, 31 October.

Roeder, Oliver (2018) ‘We Gave You 3 Million Russian Troll Tweets. Here’s What You’ve Found So Far’, FiveThirtyEight, 8 August.

Roermund, Jannes van (2017) ‘Wie Zijn de Mensen Achter Het Nepnieuws in Nederland?’ Nieuwe

Revu, 26 April. Roig-Franzia, Manuel (2019) ‘Inside the Spectacular Fall of the Granddaddy of Right-Wing

Conspiracy Sites’, The Washington Post, 2 April.

 190 https://www.washingtonpost.com/lifestyle/style/inside-the-spectacular-fall-of-the-granddaddyof-right-wing-conspiracy-sites/2019/04/02/6ac53122-3ba6-11e9-a06c-

3ec8ed509d15_story.html?utm_term=.7032f27e6939. Rogers, Richard (2019) Doing Digital Methods, London: Sage. ______ (2018a) ‘Otherwise Engaged: Social Media from Vanity Metrics to Critical Analytics’,

International Journal of Communication, 12: 450-472. ______ (2018b) ‘Digital Methods for Cross-Platform Analysis’, in Jean Burgess, Alice Marwick and

Thomas Poell (eds.), SAGE Handbook of Social Media. London: Sage, pp. 91-110. ______(2017) ‘Foundations of digital methods: Query design’ in Mirko Tobias Schäfer and Karin

Van Es (eds.) The Datafied Society. Studying Culture through Data, Amsterdam: Amsterdam University Press, pp. 75-94.

______ (2013) Digital Methods, Cambridge, MA: MIT Press.

______(2013). Debanalizing Twitter: The transformation of an object of study, in Proceedings of the 5th Annual ACM Web Science Conference, New York, NY: ACM, pp. 356-365.

______ (2005) ‘Poignancy in the US political blogosphere’, Aslib Proceedings: New Information

Perspectives, 57(4): 356-368. Romero, Jessica (2019) ‘Preventing Inauthentic Behavior on Instagram’, Facebook Newsroom, 25 April.

https://newsroom.fb.com/news/2019/04/preventing-inauthentic-behavior-on-instagram/ Rone, Julia (2019) ‘Why talking about “disinformation” misses the point when considering radical

right “alternative” media’, LSE Media Policy project blog, 9 January. Rony, Md Main Uddin, Naeemul Hassan and Mohammad Yousuf (2017) “BaitBuster: Destined to

Save You Some Clicks”. Proceedings of Computation+Journalism Symposium, Northwestern University, October.

Roozenbeek, Jon and Sander van der Linden (2018) ‘The fake news game: actively inoculating against the risk of misinformation’, Journal of Risk Research, DOI:

10.1080/13669877.2018.1443491. RTL Nieuws (2018) ‘Alles Over Complottheorie QAnon - #TrumpUpdate 77’, RTL Nieuws, 15

September. https://www.rtlnieuws.nl/nieuws/laatste-videos-nieuws/video/4417801/alles-overcomplottheorie-qanon-trumpupdate-77.

Ruusuvirta, Outi (2010) ‘Much Ado About Nothing? Online Voting Advice Applications in Finland’, in Cedroni, Lorella and Diego Garzia (eds.), Voting Advice Applications in Europe. The State of the

Art, Napoli: ScriptaWeb, pp. 47-77. Sängerlaub, Alexander, Miriam Meier and Wolf-Dieter Rühl (2017) ‘Fakten statt Fakes: Das

Phänomen “Fake News”’, Berlin: Stiftung Neue Verantwortung. Shane, Scott (2018) ‘How Unwitting Americans Encountered Russian Operatives Online’, New York

Times, 18 February. Shao, Chengcheng, Giovanni Luca Ciampaglia, Onur Varol, Kai-Cheng Yang, Alessandro Flammini

and Filippo Menczer (2018) ‘The spread of low-credibility content by social bots’, Nature Communications, (9)4787, 20 November.

Shaw, Eugene (1979) ‘Agenda-Setting and Mass Communication Theory’, International Communication Gazette, 25(2): 96–105.

Shead, Sam (2019) ‘Facebook Reacts to Live-Streamed Footage of the Deadly New Zealand Mass

Shooting That Was Posted on Its Platform’, Business Insider Australia, March 15.

 191 https://www.thisisinsider.com/facebook-responds-to-alleged-live-stream-of-christchurchshooting-2019-3.

Shieber, Jason (2017) ‘How Reports from 4chan on the Las Vegas Shooting Showed Up on Google Top Stories’, TechCrunch (blog), 2 October. http://social.techcrunch.com/2017/10/02/howreports-from-4chan-on-the-las-vegas-shooting-showed-up-on-google-top-stories/

Silverman, Craig, J. Lester Feder, Saska Cvetkovska and Aubrey Belford (2018) ‘Macedonia’s Pro

Trump Fake News Industry Had American Links, And Is Under Investigation For Possible Russia Ties’, Buzzfeed News, 18 July.

Silverman, Craig (2016) ‘This Analysis Shows How Viral Fake Election News Stories Outperformed Real News On Facebook’, Buzzfeed News, 16 November.

Silverman, Craig and Lawrence Alexander (2016) ‘How Teens In The Balkans Are Duping Trump

Supporters With Fake News’, Buzzfeed News, 3 November. Silverman, Craig and Jeremy Singer-Vine (2016) ‘The True Story Behind The Biggest Fake News Hit

Of The Election’, Buzzfeed News, 16 December. Silverman, Craig, Lauren Strapagiel, Hamza Shaban, Ellie Hall and Jeremy Singer-Vine (2016)

‘Hyperpartisan Facebook Pages Are Publishing False And Misleading Information At An Alarming Rate’, Buzzfeed News, 20 October.

Sismondo, Sergio (2017) ‘Post-truth?’, Social Studies of Science, 47(1): 3–6.

Snyder, Timothy (2018) The Road to Unfreedom: Russia, Europe, America, New York, NY: Tim Duggan Books.

Solon, Olivia and Sam Levin (2016) ‘How Google’s search algorithm spreads false information with a rightwing bias’, The Guardian, 16 December.

Sommer, Will (2018) ‘Instagram Is the Alt-Right’s New Favorite Haven’, The Daily Beast, 30 October.

Southern Poverty Law Center (n.d.), ‘WorldNetDaily’, blog, Southern Poverty Law Center. https://www.splcenter.org/fighting-hate/extremist-files/group/worldnetdaily.

Srnicek, Nick (2017) Platform Capitalism. Cambridge: Polity Press.

Statista (2019) ‘Most famous social network sites worldwide as of January 2019, ranked by number of active users (in millions)’, Statista. https://www.statista.com/statistics/272014/global-socialnetworks-ranked-by-number-of-users/

Still, Keith (2017) ‘Inauguration Crowd Size’, Crowd Safety and Risk Analysis blog, Manchester:

Manchester Metropolitan University. http://www.gkstill.com/CV/Projects/Counting.html. Swedish Civil Contingencies Agency (2018) Countering information influence activities: A handbook for

communicators, Karlstad: Swedish Civil Contingencies Agency (MSB). Swedish Media Council (2019) Källkritik — en utmaning, webpage,

https://statensmedierad.se/larommedier/kallkritikvemvadvarfor/kallkritikenutmaning.422.html, 5 March.

Tan, Yvette (2017) ‘There's a vending machine selling fake Instagram likes, because this is what we've become’, Mashable, 7 June.

Tandoc Jr., Edson C., Zheng Wei Lim and Richard Ling (2018) ‘Defining “Fake News”: A typology of scholarly definitions’, Digital Journalism, 2: 137-153.

Thomas, David R. (n.d.) ‘A General Inductive Approach for Analyzing Qualitative Evaluation Data’, American Journal of Evaluation 27 (2): 238–46. https://doi.org/10.1177/1098214005283748.

Timberg, Craig (2017) “Russian propaganda may have been shared hundreds of millions of times, new research says”. Washington Post, 5 October.

 192 Tripodi, Francesca (2018) ‘Alternative Facts, Alternative Truths’, Data & Society: Points blog, 23

February. Tucker, Joshua A., Andrew Guess, Pablo Barberá, Cristian Vaccari, Alexandra Siegel, Sergey

Sanovich, Denis Stukal, and Brendan Nyhan (2018) ‘Social Media, Political Polarization, and Political Disinformation: A Review of the Scientific Literature’, Report, Menlo Park, CA: William and Flora Hewlett Foundation.

Tufekci, Zeynep (2018) ‘YouTube, the Great Radicalizer’, The New York Times, 10 March. https://www.nytimes.com/2018/03/10/opinion/sunday/youtube-politics-radical.html.

Turner, Stephen (2001) ‘What is the problem with experts?’, Social Studies of Science, 31(1): 123–149.

Tuters, Marc, and Daniël De Zeeuw (2019) ‘Teh Internet Is Serious Business: On the Deep

Vernacular Web Imaginary’, ANSOC Working Paper. Tuters, Marc, Emilija Jokubauskaitė, and Daniel Bach (2018) ‘Post-Truth Protest: How 4chan

Cooked Up the Pizzagate Bullshit’, M/C Journal 21(3). http://journal.mediaculture.org.au/index.php/mcjournal/article/view/1422.

Tuters, Marc, Emilija Jokubauskaitė and Daniel Bach (2018) ‘Post-Truth Protest: How 4chan

Cooked Up the Pizzagate Bullshit’, M/C: A Journal of Media and Culture, 21(3). Vaidhyanathan, Siva (2018) ‘Why Facebook Will Never Be Free of Fakes’, New York Times, 5

September. Vaidhyanathan, Siva (2017) ‘Facebook Wins, Democracy Loses’, New York Times, 8 September. Vassil, Kristjan (2011) Voting smarter? The impact of voting advice applications on political behavior, PhD

dissertation, European University Institute, Tartu, Estonia. Venturini, Tommaso (2019) ‘From Fake to Junk News, the Data Politics of Online Virality’, in

Didier Bigo, Engin Isin, and Evelyn Ruppert (eds), Data Politics: Worlds, Subjects, Rights. London: Routledge. https://hal.archives-ouvertes.fr/hal-02003893.

Volkova, Svitlana, Kyle Shaffer, Jin Yea Jang and Nathan Hodas (2017) ‘Separating facts from fiction: Linguistic models to classify suspicious and trusted news posts on Twitter’, in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2: 647-653.

Volpicelli, Gina (2019) ‘This is how Facebook’s news feed fact-checking will work in the UK,’ Wired, 11 January.

Vosoughi, Soroush, Deb Roy and Sinan Aral (2018) ‘The spread of true and false news online’,

Science, 359(6380): 1146-1151. Walgrave, Stefaan, Michiel Nuytemans and Koen Pepermans (2009) ‘Voting Aid Applications and

the Effect of Statement Selection’, West European Politics, 32(6): 1161-1180. Wang, Rong, Wenlin Liu, and Shuyang Gao (2016). ‘Hashtags and information virality in networked

social movement: Examining hashtag co-occurrence patterns’, Online Information Review, 40.7: 850- 866.

Wardle, Claire (2018) Information Disorder: The Essential Glossary, Shorenstein Center, Cambridge, MA: Harvard Kennedy School.

Wardle, Claire and Hossein Derakhshan (2017) Information Disorder: Toward an interdisciplinary framework for research and policy making, Strasbourg: Council of Europe, September 27.

Waters, Richard (2017) ‘Facebook and Google help showcase Las Vegas fake news’, Financial Times, 3 October.

Weisman, Jonathan (2018) (((Semitism))): Being Jewish in America in the Age of Trump, New York, NY: St. Martin's Press.

 193 Weiss, Ben (2018) ‘Opinion | Meet the Renegades of the Intellectual Dark Web’, New York Times, 8

May. https://www.nytimes.com/2018/05/08/opinion/intellectual-dark-web.html. Wendling, Mike (2018a) ‘The (almost) complete history of ‘fake news’’, BBC News, 22 January. _____ (2018b). Alt-Right: From 4chan to the White House. London: Pluto Press.

Wieringa, Maranke, Tim de Winkel and Callum Lewis (2017) ‘Wie is de waakhond op sociale media?’, report, Utrecht: Utrecht Data School.

Woolley, Samuel and Philip N. Howard (2016) ‘Social media, revolution, and the rise of the political Bot’, in Piers Robinson, Philip Seib, Romy Fröhlich (eds.), Handbook of Media, Conflict and Security. New York, NY: Routledge, 282-292.

Wynne, Brian (1991) ‘Knowledge in context’, Science, Technology & Human Values, 16(1): 111-121.

YouTube (2019) ‘Related channels discontinued May 2019’. https://support.google.com/youtube/answer/7216516?hl=en

_____ (2018) ‘Press - YouTube’ https://www.youtube.com/intl/en-GB/yt/about/press/.

Zannettou, Savvas, Tristan Caulfield, Emiliano De Cristofaro, Nicolas Kourtelris, Ilias Leontiadis,

Michael Sirivianos, Gianluca Stringhini, and Jeremy Blackburn (2017) ‘The Web Centipede:

Understanding How Web Communities Influence Each Other through the Lens of Mainstream

and Alternative News Sources’, 17th ACM Internet Measurement Conference, 405–17.

van Zijl, Frank and Huib Modderkolk (2017) ‘Generaal: Nederland kan digitale dreiging niet aan’, De

Volkskrant, 29 August.

van Zoonen, Liesbet, et al. (2011) ‘YouTube Interactions between Agonism, Antagonism and

Dialogue: Video Responses to the Anti-Islam Film Fitna’, New Media & Society, 13(8): 1283–300.

DOI:10.1177/1461444811405020.

 194