Mediation Analysis in Communication Research: Trends and Characteristics

Mediation analysis in communication science: Examining the study of indirect effects in communication journals between 1996-2017 Top Paper Award and Top Method Paper Award (2019) Communication Theory and Methodology Division (AEJMC) MICHAEL CHAN, PANFENG HU, MACAU MAK Mediation analysis and warranted inferences in media and communication research: Examining research design in the field's prominent journals Top Paper Award and Top Method Paper Award (2020) Communication Theory and Methodology Division (AEJMC) MICHAEL CHAN, MACAU MAK, PANFENG HU 1

Mediation analysis One of the most popular methods used to test theoretical models and causal processes in communication research M M1 X Y X Y Simple mediation model (1 indirect effect) M2 Multiple mediation model (2 indirect effects) Elaborative processing News attention M1 M2 M1 X M3 Y X Y Serial mediation model (3 indirect effects) Political knowledge M2 Surveillance gratifications Cognitive mediation model of learning from the news (Eveland, 2001) Multiple and serial mediation model (5 indirect effects)

Popularity Top 100 cited papers of all time in Web of Science in the past 100 years (van Noorden, Maher & Nuzzo, 2014) Source: https://www.nature.com/news/the-top-100-papers-1.16224

Relevance and research gaps Many media and communication theories and frameworks propose indirect effects (Holbert & Stephenson, 2003) Mediation analysis provides a conceptual and statistical tool for researchers to understand how and why media effects occur (Valkenburg et al., 2016) Helps us explicate and test the processes, mechanisms and conditions in which X affects Y However, previous reviews of mediation in communication have become quite dated. E.g.: (Holbert & Stephenson, 2002; Preacher & Hayes, 2008; Slater & Gleason, 2012) Personal interest and experiences (2019) RQ. What are the characteristics and trends of mediation/indirect effects analyses in the past two decades in communication journals?

Sampling Full texts articles of 14 journals downloaded from Academic Search Ultimate: Communication Source and Communication Studies: A SAGE Full- Text Collection Keywords: mediation , mediating effect , and indirect effect . Time period = 1/1/1996 to 12/31/2017 Result: 649 full-text articles downloaded Final sample = 595 articles Asian Journal of Communication Communication Monographs Communication Research Human Communication Research International Journal of Press/Politics Journal of Broadcasting & Electronic Media Journal of Communication Journal of Computer-Mediated Communication Journalism & Mass Communication Quarterly Mass Communication & Society (1998-) Media Psychology (1999-) New Media & Society (1999-) Political Communication Public Opinion Quarterly

Growth in number of published articles that use mediation analysis (N = 595) Four journals accounted for 57% of all studies: CR (27%), JoC (11%), JMCQ (10%), HCR (9%) Mediation analysis comprised 2.8% of total published articles in the four journals in 1996. It was 25.8% in 2017

Tools/software used for mediation analysis in proportion to published articles Preacher & Hayes macros Single mediator models (2004) N = 61 Multiple meditator models (2008) N = 79 Haye s PROCESS macro (2012/2013) N = 86

2020 study The 2019 paper focused on the statistics behind mediation analysis. The 2020 paper focused on research design issues with mediation analysis Theoretical reasoning Research design Statistical analysis It is the design, not the statistical method, that permits causal hypotheses to be adequately tested (Bullock et al., 1994, p. 254) Other fields have addressed research design for mediation, including political science (Bullock et al., 2011), social psychology (Fiedler et al., 2011) and organizational and management sciences (Stone-Romero & Rosopa, 2007). Not so in communication.

Establishing causal inference Causality and experiments Three requirements to establish causality between X and Y 1.Observable association 2.Temporal order 3.Non-spuriousness U2 U3 M X Y X Y U1 U1

Alternative models Testing simple mediation using cross-sectional studies Same data can be explained in many other ways Fiedler, K., Harris, C., & Schott, M. (2018). Unwarranted inferences from statistical mediation tests An analysis of articles published in 2015. Journal of Experimental Social Psychology, 75, 95-102

Sample 1. Research design used in the study E = Experiment O = Observational L = Longitudinal M = Mixed I = Independent variable M = Mediator B = Both 1 = No 2+ = Yes Si = Simple mediation Si+M = Moderated simple mediation P = Parallel mediation P+M = Moderated parallel mediation Se = Serial moderation C = Complex mediation N = No Y = Yes N = No Yg = Yes, but does not mention which mediators Ys = Yes, and names specific mediators N = Not mentioned NS = Not mentioned as there were no statistically significant mediation findings Y-now = In the present tense Y-past = In the past tense N = No Y = [insert limitation] Sample and codebook Used dataset of previous paper, focusing on Journal of Communication, Communication Research, Human Communication Research, and Journalism & Mass Communication Quarterly (N = 332). 2. If experiment, which variable(s) is/are manipulated? 3. Does the article contain multiple studies? 4. Type of mediation model tested 5. In the results section, are alternative models tested and reported? In the discussion section, is the possibly of other mediators that can plausibly mediate the proposed X Y relationship(s) acknowledged or proposed? 6. Three coders. Still calculating reliability 7. In the discussion section, is the significant effect of the mediator explicated in the past or present tense? 8. In the discussion section, are limitations of the research designs acknowledged?

Findings Observational designs = 47% (N = 157) Experiment design = 41% (N = 137) Longitudinal design = 11% (N = 35) Multiple designs = 1% (N = 4).

Findings Of 157 observational studies, 65% (N = 102) adopted parallel, serial and complex mediation designs, which limits the plausibility of the models being the correct model because the variables were measured at the same time

Findings Experiments Manipulate Independent Variables but Not the Mediators All adopt the measurement-of-mediation experiment design National identification Perceived discrimination Hostility towards society Anti-muslim ads Perceived discrimination National identification Self-esteem Anti-muslim ads Self-esteem Schmuck, D., Matthes, J., & Paul, F. H. (2017). Negative Stereotypical Portrayals of Muslims in Right-Wing Populist Campaigns: Perceived Discrimination, Social Identity Threats, and Hostility Among Young Muslim Adults. Journal of Communication, 67(4), 610-634. Hostility towards society Only 22 studies (16%) acknowledged the correlational nature of the M Y relationship that inhibits the ability to draw concrete causal inferences.

Writing about limitations Observational studies Limited discussion of research design decisions and implications. Acknowledged in only 59% of observational studies: (1) The data was cross-sectional (2) Causal claims cannot be made on the direction of the effects (3) Findings are based on strong theoretical foundations and/or past research. (4) Future longitudinal or experimental studies are needed to verify causality of the relationships found in the current study. A few observational studies (14%) attempted to strengthen their causal claims by testing alternative models against the data.

Unwarranted inferences? Past tense Past .v. Present tense 41% Past tense Causal inferences confined to findings within the study This study s results extended the stress assumption by revealing that both brokering frequency and negative brokering feelings exhibited indirect effects on alcohol use and other risky behaviors through family-based acculturation stress (Kam, 2011, p. 468) For those high on search for meaning, boundary expansion mediated an effect of self- affirmation not only on enjoyment, appreciation, suspense, and transportation, but also on fun and identification (Johnson et al., 2016, p. 402). Our results also found indirect effects of information seeking on future information seeking via skepticism (Hutchens et al., 2016, p. 1085). 53% Present tense Causal inferences as a generalizable law Present tense Police reality show viewing retains strong and consistent direct relationships with all subsequent variables in the model. In addition, this type of TV viewing generates dual indirect associations with attitudes toward capital punishment and guns, respectively (Holbert et al., 2004, p. 355). Results show that relational uncertainty and partner interference (a) are unique predictors of turbulence beyond relationship satisfaction, and (b) partially mediate the link between relationship satisfaction and turbulence (Theiss & Knobloch, 2014, p. 45). This study identifies a political-communication process whereby fake-news viewing impacts on political attitudes, enhancing the feelings of inefficacy, alienation, and cynicism toward politicians. This relationship is mediated by the perceived realism of fake news (Balmas, 2014, p. 446). 6% No claim

Unwarranted inferences? No experimental manipulation of mediator(s) Experiments Manipulate Independent Variables but Not the Mediators This claim, however, should be stated somewhat tentatively and confirmed with additional research because goal frequency and appropriateness were not directly manipulated (Palomares, 2013, p. 92). Furthermore, we did not explicitly manipulate emotions in our design, and thus strictly we cannot make valid inferences about the role of emotions as a causal mechanism of framing effects (Powell et al., 2015, p. 1012) 16% (N = 22) of studies acknowledged correlational nature of the M Y relationship Correlational relationship between mediator(s) and dependent variable Third and most important, because we measured anticipated regret and intentions at the same time points, the mediational analyses are correlational and limit our certainty about the causal order (van Koningsbruggen et al., 2016, p. 1039) Moreover, although we used an experimental approach to establish a causal relationship between political ad exposure and perceived discrimination, we measured all dependent variables at one point of time, which strictly speaking only allows correlational evidence for the mediators and dependent variables (Schmuck et al., 2017, p. 626). Both explanations Also, it should be noted that although one can make a strong causal inference about the direct effect of slant, a manipulated factor, the relationships between personal opinion and perceptions are correlational and must be interpreted with more caution (Gunther, 1998, p. 498).

Unwarranted inferences? Alternative mediators are rarely acknowledged or proposed 14% (N = 22) observational studies proposed alternative mediators that can also explain the X Y relationship General call for alternative mediators to be examined It should be noted that our model, although well-supported empirically, is only one candidate among many possible mediating mechanisms. As some scholars have urged, more effort should be made to find and refine mediating mechanisms of this sort (Paek & Gunther, 2007). Future work should also examine additional potential mediators particularly those that are more likely to display effects of actually listening to music (Harwood et al., 2016). Alternative mediators specified This study focused on testing affective mediators, thereby neglecting the likely interplay of emotions with cognitive mediators such as changes in issue accessibility or applicability (Lecheler et al., 2015). Future work should extend to more diverse samples and capture persuasion processes in more nuanced fashion, for example, by looking at absorption, funniness perceptions, and so forth (Boukes et al., 2015) as mediators (Knobloch-Westerwick & Lavis, 2017). 18% (N = 24) for experimental studies 23% (N = 8) for longitudinal studies

State of mediation Critics in other fields and disciplines Most of these studies are based on designs or analysis strategies that are inadequate to establish mediation, so relatively little of the extant literature on mediation is actually worthwhile (Kline, 2015) Much of the knowledge generated in top journals is likely false (Saylors & Trafimow, 2020) Summary of our findings Extant research designs are inadequate to draw strong causal inferences. Authors were generally quite eager to claim that their significant mediation models can be generalized to the real world but pay less attention to acknowledging limitations of their research designs and the possibility of other plausible mediators that can also account for the proposed mechanisms.

State of mediation Symptoms Theory. Pervading indirect effects paradigm in the field that places emphasis on explicating mechanisms that underlie media effects Editorial demands. Journal editors and reviewers expect studies on mechanisms and processes (the why ) Its easy. Anyone with a rudimentary understanding of regression statistics can run mediation analysis, leading to more studies based on ad-hoc data discovery Norms. If mediation studies based on cross-sectional surveys and experiments that only manipulate the independent variable are perceived to be sufficient for publication in the field s top journals, then researchers may consider such designs to be enough Education. Lack of methods training or awareness in graduate school curriculums

Communication mediation State of mediation A half-glass full perspective Practically, insistence on watertight research designs for mediation analysis will impede the progress of knowledge creation and dissemination. Shah, D. V., et al. (2007). Campaign Ads, Online Messaging, and Participation: Extending the Communication Mediation Model. Journal of Communication, 57, 676-703. Mediation studies that utilize less than adequate designs can be sources of new ideas, perspectives, and propositions that serve as springboards for further exploration using more rigorous research designs. Shehata, A., & Amn , E. (2019). The Development of Political Interest Among Adolescents: A Communication Mediation Approach Using Five Waves of Panel Data. Communication Research, 46(8), 1055-1077.

Recommendations Improve existing practices Temper claims of causal inference even when their mediation models are statistically significant . Acknowledge that the data can potentially account for other alternative models or mechanisms. Acknowledge other possible unmeasured mediators that can account for the X Y relationship Provide substantive recommendations on how the points above can be achieved. E.g.: Therefore, it is crucial that future studies replicating our results establish a temporal order between anticipated regret and intentions in order to more precisely follow the causal chain approach assumed when testing mediation, (van Koningsbruggen et. al., p. 1039).

Recommendations Manipulation of mediator designs for stronger causal inferences Double randomization design (Pirlott & MacKinnon, 2016), also known as the experimental-causal-chain design (Spencer et al., 2005) and two randomized experiments design (Stone-Romero & Rosopa, 2007). Study 1 X M Y X(M) Y Study 2 Concurrent double randomization design (Pirlott & MacKinnon, 2016) Manipulate both X and M in a 2-factor experiment.

Thank you! Michael Chan Hu Panfeng Macau Mak