The Rowe Replication Study
Read the full article here
Rowe (2015) studied the effects of anonymity on commenting on news articles. He compared comments on articles on the Washington Post website and the Washington Post Facebook page and found that 6% of the website comments and 2.7% of the Facebook comments were uncivil. As the results in the previous chapter show, and as repeated in Table 5.1, my study found that 8.7% of the comments on vg.no and 3.4% of comments on VG’s Facebook page were derogatory. And my own coding of comments from the Washington Post, the same newspaper that Rowe used for his research, resulted in 7% of comments from the Washington Post comment section and 3,7 from its Facebook page being coded as derogatory.
Rowe explains the difference he found in civility between the two platforms with anonymity, arguing that there are more uncivil comments on the Washington Post comment section because the commenters there are anonymous. The Washington Post uses their own comment system where users have to sign in to comment. To create an account, the user can either use an amazon account, Facebook account, or their e-mail. Either way, when commenting, the user is only identified through a pseudonym. This allows commenters on the Washington Post to be anonymous. On Facebook, however, commenters are not anonymous. Using anonymity to explain Rowe’s results may seem like a reasonable explanation. But I will argue that this explanation is inaccurate because my own results shows a higher number of derogatory comments in the website set – just like Rowe’s results. But if Rowe’s interpretation about anonymity as the main explanatory factor was correct, one would expect the number of derogatory comments on vg.no and VG’s Facebook page to be similar, because neither of these two platforms allow for anonymity.
Because Rowe used a very different coding scheme than mine, it’s difficult to make any conclusions. But because I already had the comments from my original research formatted and anonymized, I decided to perform a new study using the same data. This new study, which I from now on will call the “Rowe replication study”, hypothesized that there would be a close to equal number of uncivil comments on the comment sections on VG’s website and their Facebook page because VG’s comment section does not allow for anonymity. Because of the similar number I had already observed between Rowe’s results and my own, I expected this hypothesis to found invalid.
The hypothesis for this study is that there will be an equal, or close to equal, number of uncivil comments on VG’s comment section and VG’s Facebook page, because commenters are not anonymous on either platform. This study aims to recreate Rowe’s study, using a different sample of data taken from two platforms where users are not anonymous. Therefore, this study replicates Rowe’s research methodology, using content analysis to code comments and compare them.
Because the “Rowe replication study” studies the comments that were collected for my main thesis study, it is important to note that there are some differences between Rowe’s study and this one in how the sample was selected. Rowe used constructed week sampling to generate a stratified sample of political news articles over two constructed weeks (2015: 127). The current study has not used this method. Rowe only studied comments on political articles, whereas this study has analyzed comments from different types of articles, including national and international politics, crime, sports and finance. Although it is not written specifically, one can assume that Rowe chose only political articles because he wanted to study political discussion in comment sections to look for incivility. I would argue that uncivil and impolite comments can be found on articles covering a wide range of topics, and that narrowing down the area of research to only cover political articles is a mistake. Another difference between Rowe’s study and this one, is that in Rowe’s study, 4502 comments from the Washington Post and 2304 comments from their Facebook page, were collected. Of these, a random sample of 500 comments from each platform was analyzed. In the current study, all collected comments have been analyzed in their original context. When doing qualitative research, I believe it is important to analyze each comment as part of a larger context. Consider the following example: “Your contribution to Norwegian industry will probably last for generations. Thanks.” This comment, which was coded as sarcastic, is only sarcastic when read in context. On its own it could potentially be a sincere message of appreciation to another commenter, or the subject being reported on in the article. By not analyzing comments in their context, the meaning of some comments may be lost on the researcher.
This study is designed to replicate the coding scheme used by Rowe. Rowe’s coding scheme is an adaptation of a pre-existing coding scheme by Papacharissi that is used to code comments as uncivil or impolite (Rowe, 2015: 128). Incivility is defined by Papacharissi as “a set of behaviors that threaten democracy, deny people their personal freedoms, and stereotype social groups” (Papacharissi, 2004: 267; Rowe, 2015: 128). Based on this, a three-item index has been developed to determine if a comment has violated the standards of democratic discourse. A comment was coded as uncivil if it verbalized a threat to democracy, threatened the rights of other individuals, or assigned stereotypes. A second index was developed to determine if a comment is impolite. A comment was coded as impolite if it contained name-calling, aspersions, claims about lying, vulgarity, pejorative speech, hyperbole, non-cooperation or sarcasm. A final category, called “other, was created for impolite comments that did not fit into any of the categories.
Most of the comments analyzed in this study were neither uncivil or unpolite, which is in line with Rowe’s results and previous research (Rowe, 2015: 129). 8.1% of the comments on VG’s website were coded as uncivil, with most of them being threats to individual rights or the use of stereotypes. Only 2.1% of the comments on VG’s Facebook page were coded as uncivil, all of them being use of stereotypes. This closely matches the results of Rowe’s coding of comments from the Washington Post. He found that 6% of the comments on the Washington Post and 2,7% of the comments on its Facebook page were uncivil.
Rowe's results have been replicated. But because commenters on vg.no are not anonymous, Rowe's explanation of anonymity causing incivility can not be confirmed by this studhy.