Uncovering Questionable Research Practices in Science

Questionable practices and misconduct in science Marcel A.L.M. van Assen Tilburg University & Utrecht University Department of Methodology and Statistics & Department of Sociology 1

Who believes in the storybook image of the scientist? (Veldkamp et al., 2017) objective, rational, open-minded, intelligent, integer, cooperative 2

Who believes in the storybook image of the scientist? (Veldkamp et al., 2017) Lay people and scientists attributed considerably more objectivity, rationality, open-mindedness, intelligence, integrity, and cooperativeness to scientists than other highly educated people Really? Scientists are mere humans, and humans are fallible 3

Main messages/story (1) Current science is about results, process leading to results is largely hidden (black box) (2) Rewards for novel-nice-significant research (3) Questionable research practices (qrps) prevalent (4) Qrps and fraud largely undetected and not punished (5) Qrps and fraud are rational! (6) Science needs to be fundamentally restructered to limit problems (including qrps and fraud) 4

Overview (1) How science works (2) Definitions of qrps and fraud (3) Qrps and their prevalence (4) Fraud and prevalence (5) Qrps and fraud are rational! (6) Restructuring science 5

(1) How science works Which papers get published in journals? Something novel Nice story (good to read) Statistically significant result Note: all three are irrelevant to science Strong evidence on all three 6

(1) How science works Novel Editor: We don t accept replication studies Nice story HARKing (Kerr 1998) Editor/reviewer: Leave out/add this/that to obtain nicer story Significant Publication bias Failures to replicate 7

(1) How science works Publication bias Fanelli (2010) Varies across disciplines UK < (rest) Europe < US < Japan 8

(1) How science works What do we see of process leading to papers? Initial theory/hypotheses? Complete methods? Data? All analyses? Distinction confirmatory/exploratory analyses? Review process? Final paper are lucky or rich) no no no no no No yes (if you 9

(1) How science works Conclusion Current science is a black box (omission process), focused on novel-nice-significant (irrelevant) 10

(2) Definitions of qrps and fraud What is fraud ? No consensus Fraud specific meaning in law scientific misconduct preferable terminology (Resnik 2012) Intention to deceive is key (Fanelli 2009) Scientific misconduct (1) Fabrication: invention data (2) Falsification: willful distortion data or results (3) Plagiarism: copy without attribution 11

(2) Definitions of qrps and fraud Questionable research practices (NAS 92) = .. actions that violate traditional values of the research enterprise and that may be detrimental to the research process.. Rather vague (list of behaviors) Working definition: detrimental to the research process without qrps results would have been different, or interpretation of results would have been different 12

(2) Definitions of qrps and fraud Questionable research practices Results and interpretation of results depend on how data were collected, (how many) analyses were performed Patients in the treatment group score significantly higher on X than patients in the control group (p = .012) p-value only correct in case of ONE test 13

(2) Definitions of qrps and fraud Questionable research practices: final comments Distinction fraud and qrps: intention to deceive (really?) Distinction and definition fraud, fabrication, falsification, QRP unclear and arbitrary (Fanelli 2009) Many shades of grey between good conduct and fraud 14

(3) Qrps and their prevalence (A) Questionnaires (B) P-values (C) Process to article (A) Questionnaires Do researchers use qrp and find them defensible? John et al. (2012, Tijdink et al. (2014), Bedeian et al. (2010), Feld et al. (2012) Questionnaires research is disputed (Fiedler & Schwarz, 2015) 15

(3) Qrps and their prevalence (A) Questionnaires Do researchers use qrp and find them defensible? Using anonymous elicitation procedure with incentives Survey > 2,000 psychologists Self-serving bias: own = collaborators < researchers own institute < researchers other institute 16

Adapted from John, Loewenstein, and Prelec (2012)

(3) Qrps and their prevalence (B) P-values Hartgerink et al (2016) Inspect p-values reported as p= .05 67.5% > 0.05 and 32.5% lower than .05 18

(3) Qrps and their prevalence (C) Process to paper Chan et al (2004) 19

(3) Qrps and their prevalence (C) Process to paper But see also O Boyle et al (2014) Tracking differences between PhD theseses and articles resulting from them Proquest Dissertations and Thesises (Management Sciences) 142 studies, 2,311 hypotheses 20

(3) Qrps and their prevalence Conclusions Qrps highly prevalent Hard to detect when science is black box or process is not reviewed 21

(4) Fraud and prevalence Prevalence Questionnaire: 0.6% (John et al 2012) Meta-analysis on questionnaires: ~2% (Fanelli) 3.8% figure manipulation (level of papers!) 22

(4) Fraud and prevalence But... Hard to detect (black box) Reviewers do not pay attention to it Scientists do not recognize it 23

(4) Fraud and prevalence Conclusions Prevalence unknown Because qrps are much more prevalent, damage to science from qrps is higher than from misconduct 24

(5) Qrps and fraud are rational! Benefits of qrps and fraud are high Helps to get nice-significant papers, at low cost (small sample sizes, quick results) Helps to obtain PhD, grants, tenure, to become a full professor Qrps are accepted or recommended by many scientists, so hardly any costs what about fraud? 25

(5) Qrps and fraud are rational! Probability of getting caught is small Undetected (black box, lack of attention, lack of expertise) 26

(5) Qrps and fraud are rational! If caught, then unlikely to be punished For many examples, see Ontspoorde wetenschap or Derailed science by Van Kolfschooten Investigation in secrecy (if any) Investigation amateuristic fraudster often protected If punished, punishment = small Whistleblower punished Fraudulous research not retracted 27

(5) Qrps and fraud are rational! One specific case: Trivers (Brown) Study in Nature (2005), a lot of publicity Brown, W. M., Cronk, L., Grochow, K., Jacobson, A., Liu, C. K., Popovi , Z., & Trivers, R. (2005). Dance reveals symmetry especially in young men. Nature,438(7071), 1148-1150. Trivers detects fraud Brown, and Cronk deny Horrible treatment of Trivers, and lots of work by Trivers Finally, ten years later, study is retracted and fraud established 28

Trivers (Brown) http://roberttrivers.com/A_Case_of_Fraud_at_Rutgers.html 29

Trivers (Brown) [continue reading] 30

Cases of misconduct: from start to end Something strange is noticed about research of Suspect by someone Mostly it ends here (1) (2) Strangeness is communicated by someone to (i) Suspect Mostly it ends here; could have bad consequences for Accusor

Cases of misconduct: from start to end Strangeness is communicated bysomeone to (ii) Colleague Mostly it ends here (2) (iii) (Integrity or high ) person within organization nothing happens Accusor experiences negative consequences investigation by own institute

Cases of misconduct: from start to end (iii) (Integrity or high ) person within organization investigation by own institute (2) Investigation usually is Ad hoc Amateuristic Opportunistic favor of Accused, Accusor gets reprimanded Investigation is carried out well, by an independent committee (as in the case of Stapel)! Result of investigation is confidential, usually in

Cases of misconduct: from start to end Strangeness is communicated by someone to (iv) Integrity institute outside organization (2) Netherlands: LOWI Independent investigation Reports to organization of Accused, which determines what is going to happen US: ORI Accused (and reward) Independent investigation Can sanction Accused and organization of

Cases of misconduct: from start to end Conclusions Whistleblower has a large disadvantage Accused and Organization of Accused are protected Investigations of fraud are often ad hoc, amateuristic (and opportunistic) Highly confidential: bad for science Few retractions, reasons of retractions unknown Gap scientific and legal investigation

(5) Qrps and fraud are rational! Conclusions Expected benefits > expected costs of fraud, hence fraud is rational behavior Partly because system, qrps and fraud will continue to exist (black box) 36

(6) Restructuring science What do we see of process leading to papers? Initial theory/hypotheses? Complete methods? Data? All analyses? Distinction confirmatory/exploratory analyses? Review process? Final paper are lucky or rich) no no no no no No yes (if you 37

(6) Restructuring science 1. Theory, Method/Design Open 2. Preregistration of Paper and Method/Design R1. Pre-review Open R1. Post-review Open 3. Data collection Open Open 4. Complete paper with data-analysis (expl. and conf.) and discussion R2. Pre-review Open R2. Post-review Open 38

(6) Restructuring science 1. Theory, Method/Design Open 2. Preregistration of Paper and Method/Design R1. Pre-review Open R1. Post-review Open 3. Data collection Open Open 4. Complete paper with data-analysis (expl. and conf.) and discussion R2. Pre-review Open R2. Post-review Open 39

(6) Restructuring science Only when science is restructured such that Theory/hypotheses, method, and planned analyses are pre-registered Reviewers check consistency between paper and pre-registration (and not novel-nice-significant) All research flow is openly shared QRPs will be limited Fraud will be more likely detected 40

The End Main messages/story (1) Current science is about results, process leading to results is largely hidden (black box) (2) Rewards for novel-nice-significant research (3) Questionable research practices (qrps) prevalent (4) Qrps and fraud largely undetected and not punished (5) Qrps and fraud are rational! (6) Science needs to be fundamentally restructered to limit problems (including qrps and fraud) 41

42