Schnall, Simone

Apollo (Cambridge)

Psychologist Dr Simone Schnall (University of Cambridge) has spoken about her own personal experience of the so-called‘replication crisis’ within psychology at an event organised bythe online debating forum Edge.  Dr Schnall began her talk speaking about the complicatednature of replications in social science. She said: ‘When itcomes to social psychology it’s a little more complicatedbecause what we usually do is test a specific question withvarious different experiments… These are called conceptualreplications. Our entire literature is built on those conceptualreplications, but those are not the ones people are nowdiscussing. They’redifferent. They’re calleddirect replications.’ According to Schnall,replicating socialpsychology experiments is not as simple as in hardsciences such as medicinewhere a direct replicationcan be done simply byadministering the samedosage of a drug to apatient. She moved on to talking about her work,which looks at the linksbetween physical and moraldisgust. Pointing to a paperby David Stanley and Jeffrey Spence where theydiscuss the expectations forreplications and used computerstimulations, in which they ranexperiments thousands of times under perfect conditions withnothing but measurement error, she said: ‘Even then one gets a great variability of results. The conclusion is that any onegiven study is not conclusive. That’s why normally we do lotsof studies to see if there’s a general pattern.’Dr Schnall suggested that after it emerged that socialpsychologist Diederik Stapel had fabricated data, leading to scores of papers being retracted in 2011, it made thecommunity adopt a mind-set where people felt they needed to do things differently and look out for fraudsters and falsepositives. As well as this shift in attitude towards replication,Dr Schnall explored the lack of a system for choosing whichstudies should be selected for replication and suggested that a disproportionate number of studies had been singled out forreplication because findings may be counterintuitive if one is unfamiliar with the literature and studies appear easy toconduct. She also said there were issues with the quality of somereplication studies and the conclusions drawn from them.‘Often the way these replications are interpreted is as if onesingle experiment disproves everything that’s come before.That’s a bit surprising, especially when a finding is negative, if an effect was not confirmed. We don’t usually do that withpositive findings… we don’t say this now proves once and forall that such and such effect is real. It perhaps comes with thatidea that it intuitively seems like this is the real study becausewe repeated exactly what had been done before.’Some of the issues with this increase in direct replicationshave affected Dr Schnall: one special issue of the journal SocialPsychology (see tinyurl.com/qe5sru4), with 15 replicationpapers covering 27 earlier reported effects, went to printwithout having undergone any peer review, and one of her own findings was reported to have not been replicated by someresearchers. She added: ‘I looked at their data, looked at theirpaper and found what I consider to be a statistical problem.What was really interesting, though, was that when I alertedthe editors, they were not very interested. They were notinterested at all. In fact, they denied me the right to apublished response. I had to fight tooth and nail to get thatresponse. ‘Let’s think about it in the legal context. This is to declare a verdict on the quality of people’s work without a judge andwithout having given the people whose work is concerned anyright to even look at the verdicts, never mind to defendthemselves.’A recent paper reported several successful replications of Schnall’s work, and a re-analysis of the claimed failedreplication actually revealed the predicted effect (seetinyurl.com/nk66bhj).Meanwhile there has been a growing concern amongacademics about the legality of post-publication peer review,and the general tone of discussion surrounding their workoutside of the formal journals publishing process (see TomStafford’s Mind Hacks article on this subject:http://tinyurl.com/qcmqpg7). For example, Times HigherEducation has reported on a scientist in the US who has startedlegal action after, he claims, anonymous comments questioninghis science cost him a job offer. ERI For a video and the full transcript of Dr Schnall’s talk seetinyurl.com/k97w4jv. See also tinyurl.com/psycho0512 for ourspecial issue on the topic.8 vol 28 no 1 january 2015NEWSSpeaking out on the ‘replication crisis’The mental health research charity MQ:Transforming Mental Health has announced£1.5 million in funding awards as part of itsnew research programme PsyIMPACT. Theawards aim to support ways of providingbetter and more accessible psychologicaltreatments for common mental disorders.The four projects that have been awardedfunding include a team at King’s CollegeLondon who are developing new evidence-based psychological treatments for worry and rumination, a dominant symptom ofdepression and anxiety disorder. The fund isalso supporting the first trial of a computer-based preventative intervention at Birkbeck,University of London, for 10-month-old infantsat risk of ADHD.Cynthia Joyce, Chief Executive of thecharity, said: ‘Psychological treatments helpso many people. Research in this area has thepotential to transform the scale and impact ofpsychological therapy in the UK, meaningmore people can get access to care in a waythat works for them.’ ERTRANSFORMING RESEARCH FUNDINGSimone Schnall

Speaking out on the 'replication crisis'

http://www.repository.cam.ac.uk/handle/1810/246526

Speaking out on the 'replication crisis'

Abstract

Similar works

Full text

Available Versions

Apollo (Cambridge)