幸运飞艇计划

This聽article by Jon K氓re Time was published in Morgenbladet No. 22/8鈥�14 June 2018, and is republished here by the courtesy of Morgenbladet. Translated from the Norwegian by Heidi Sommerstad 脴deg氓rd, AMESTO/Semantix Translations Norway AS. The original article in Norwegian is availiable 聽(behind paywall).聽

鈥淭here鈥檚 been a lot of data torturing with this cool dataset,鈥� wrote Cornell University food scientist Brian Wansink in a message to a colleague.聽

Wansink was famous, and his research was popular. It showed, among other things, how to lose weight without much effort. All anyone needed to do was make small changes to their habits and environment. We eat less if we put the cereal boxes in the cabinet instead of having it out on the counter. But after an in-depth investigation by Buzzfeed earlier this year, Wansink has fallen from grace as a social science star to a textbook example of what respectable scientists do not do.聽

鈥淒ata torturing鈥� is also referred to as 鈥減-hacking鈥� 鈥� the slicing and dicing of a dataset until you find a statistical context that is strong and exciting enough to publish. As Wansink himself formulated it in one of the uncovered emails, you can 鈥渢weak鈥� the dataset to find what you want. 鈥淭hink about all the different ways you can cut the data,鈥� he wrote to a new grad student about to join his lab.聽

It鈥檚 a crisis! 鈥淚s science in crisis? Well, it was perhaps debatable a few years ago, but now?鈥澛�

Andrea Saltelli seems almost ready to throw up his hands in despair at the question. He鈥檚 an Italian chemist and statistician with a background as the head of the European Commission鈥檚 鈥淓conometrics and Applied Statistics Unit.鈥� Today, he鈥檚 a guest researcher at the Centre for the Study of the Sciences and the Humanities at the University of Bergen. Lately, he specialises in what he believes are serious problems with the reliability of science, especially how the statistics and research used to justify political decisions, are, on closer scrutiny, often more uncertain and less impartial than the impression we are given.聽

The issue is recognisable in Norwegian social debates. Should we trust Fisheries Minister Per Sandberg when he, backed by the Institute of Marine Research and the Norwegian Food Safety Authority, says we can safely eat farmed salmon? Is Statistics Norway鈥檚 鈥渋mmigration聽account鈥� a suitable basis for policy? And is it really as simple as the research shows that more teachers don鈥檛 have an 鈥渆ffect鈥�?聽

Issues of this nature are what Saltelli and his colleagues discuss in the book Science on the Verge (2017). In the book, Saltelli draws a picture of a serious and comprehensive system crisis.聽

This statistician from Rome has levelled withering criticism, claiming in a lecture that 鈥渢he pathologies鈥� exhibited by science can be likened to the traffic in indulgences that fomented Martin Luther鈥檚 rage against the Roman Catholic Church in the 1500s.聽

The trouble with marshmallows. But hey, wait a minute! How can we talk about a 鈥渃risis鈥� at the dawn of the gene-editing age? When scientists create enzymes that eat plastic and finally begin to get the hang of robots. When children can be born with three parents. Just recently (last year!), astrophysics gained measurable evidence that Albert Einstein鈥檚 theory about gravitational waves was right. Oh yes, Saltelli answers.聽

鈥淪cience is still delivering miracles big-time. My point is that a formidable share of the research we pay for is quite simply wrong. Science has lost the ability to self-police its production, and it doesn鈥檛 seem as if society has any idea what it should do,鈥� says Saltelli.聽

Among other things, it鈥檚 about the uncertainty that has spread in recent years through parts of the world of science, which both the media and research literature have dubbed the 鈥渞eproducibility crisis鈥� or 鈥渢he replication crisis鈥�: It turns out that a lot of research is difficult or impossible for other scientists to review accurately. And when independent scientists try to reproduce previous experiments, many have been frequently amazed that the findings can鈥檛 be verified.聽

At the heart of the current debate about the crisis is psychology, where the discussion has to some degree been harrowing. The final chapter was written last week when the famous Marshmallow experiment got a shot across the bow: In this influential experiment, the self-control of four-year-olds was tested to see if they wait a short while to eat a treat. If they managed they would receive a second treat a little later.聽

A major attempt at replicating the study nevertheless indicates that the popular idea that there is a correlation between the ability of small children to exercise self-restraint and their success later in life may be false (see sidebar). But the discussion on reproducibility is also ongoing in everything from cancer research and medicine to economics and sports science.聽

This is 鈥渁 significant crisis.鈥� Fifty-two per cent of scientists agreed that this was the case when the journal Nature conducted a (controversial) interdisciplinary survey on the topic in 2016, which is often referred to in the debate. Fully 90 per cent agreed that the crisis 鈥渆xists.鈥澛�

Studies with few participants, secrecy of data, poorly designed experiments and misunderstood use of statistical analysis are some of the explanations for the phenomenon. Not least, many people point to target-oriented management and the pressure to publish in academia as the root cause: Do the reward systems create overproduction of articles and an abundance of wasted or bad research? Is it too tempting to take shortcuts to get more or more sensational articles in print?聽

Democratic problem. In Norway, biologist Dag O. Hesse expresses concern about the conditions for free research in his most recent book, Sannhet til salgs (Truth for Sale). 聽He warns against increased political control, market and utilitarian thinking and harmful聽publication points. Not to mention the explosion of rogue journals and fake science.聽According to Saltelli, everything correlates 鈥� the reproducibility crisis is just the tip of a massive iceberg.聽鈥淪cience itself is compromised,鈥� he says.聽

The poor quality control will ultimately undermine聽democracy, Saltelli claims. Modern society rests on a well-functioning marriage of science and political power, where research legitimises power by standing as a guarantor of truth.聽

鈥淒ark forces are entering because a void has been created. Society is vulnerable. It鈥檚 like a kind of ecosystem, where predators take over because other species are weakened.鈥澛�

The inexplicit message of numbers. The news went around the world in 2015: 7.9 per cent of all the species on Earth will die out due to climate change.聽

鈥淭hat鈥檚 ridiculous. For how can you predict this, right down to the decimal, when we don鈥檛 even know how many species we have today?鈥� asks Saltelli.聽

Excessive trust placed in numbers and models is part of the reason for people鈥檚 distrust of science, he believes. Mathematical modelling is being misused to create all kinds of amazing constructions: Actually, it is often the case that series of uncertain assumptions are given a numerical value and put into a model. The result the computer grinds out is taken to be fact, but the fact hides several layers of scientific uncertainty and valuations, which thus become invisible in the political debate.聽

鈥淵ou read sentences in scientific articles that read 鈥榳ith a sea temperature that is predicted to increase by 2.5 degrees by 2100...鈥� and it feels like it鈥檚 a matter of fact. But who knows? It can be 5 degrees or 1 degree or whatever. The system is so complex that any attempt at predictions over such a time span is hubris. Yet such numbers have become part of the public debate,鈥� says Saltelli.聽

Statistics Norway鈥檚 now discredited immigration account, the report that deals with the impact of immigration on the state budget up to the year 2100, may perhaps be another example: Not only is the calculation based on highly uncertain population projections, it also does not take into account that the labour market adapts to wages and prices. And the comparative basis on which the entire report is based is a very hypothetical Norway where the boundaries are closed and the population actually declines without it having any negative financial ramifications.聽

The problem is that there is uncertainty on top of uncertainty on top of uncertainty?聽

鈥淎ny sensible scientist would be very careful to make such claims. But unfortunately, scientists now use this strategically to defend their positions, to promote their agenda in public,鈥� says Saltelli.聽

Denial. There is an 鈥渁ttitude of denial鈥� to the crisis in scientific institutions and in research policy, Saltelli believes. He claims the reproducibility issue is not even mentioned in key research policy papers and documents that deal with how research should influence politics.

鈥淚f you want to use research to give advice on policy, can you ignore being concerned about the fact that science is experiencing this crisis in quality control? It makes no sense, so this behaviour is strategically defensive. The silence speaks volumes.鈥澛�

Saltelli then lets loose with: The whole notion of knowledge-based (evidence-based) policy is insane. All too often, research becomes, in practice, an instrument that legitimises predetermined political purposes. Knowledge from research becomes a currency that can be used to buy political clout, he has written.聽

鈥淧eople with deep pockets can gain control over more research-based knowledge. They can promote it more aggressively than others and use it as a framework that everyone else must deal with.鈥澛�

Never clean. Saltelli is one of several researchers at the University of Bergen who market an approach to science, technology and the use of research in society called 鈥減ost-normal science.鈥� It entails being open about uncertainty, complexity and disagreement, especially when values are at stake and the risks are great.聽

Saltelli believes that far too many researchers still cling to the idea that research is neutral and objective truth 鈥� ideas that also live on in popular culture. But he believes that more scientists now experience a conflict between this romantic vision and what he believes is the reality.聽

鈥淔or example, it could be because they work for commercial companies or have to defend their skin or career or whatever. This conflict is one of the reasons for the collapse in trust in science and its reputation.鈥澛�

Can it be fixed? After looking at how scientists use statistics, John Ioannidis at Stanford University claimed as early as 2005 that most published research findings must be incorrect. 聽Ioannidis is a leading figure in the field of meta-research, in other words research about research. He has also estimated that 85 per cent of the money that goes to biomedical research is 鈥渨asted.鈥� Since then, he has done similar critical studies on several disciplines, most recently in Nature a few weeks ago where he was one of the co-authors of an article on brain research on gender differences, where they point out systematic weaknesses in the research and conclude that gender differences may in reality be less than current scientific literature tells us.

Ioannidis is one of those who have now taken the lead in the effort to find good solutions to the problems with unreliable results. They include tightening methodologies and facilitating more open data sharing, multiple replications and pre-registration of studies. But such聽measures are not enough to deal with the 鈥渃risis,鈥� Saltelli believes. It is too extensive to be fixed by giving doctoral students courses in statistics or by making slight adjustments to publication points.聽鈥淚 have great respect and admiration for my colleagues who are working on this. There aren鈥檛 many of them and they need all the encouragement they can get. But I still believe that the rabbit hole goes deeper,鈥� says the statistician.聽How deep?聽

The monster in the rabbit hole. 鈥淚f we dig deep enough, this is about the fact that we have a quality control system in science that was appropriate when research was carried out in smaller communities of experts. Today, however, we can talk about mega science that is largely aimed at promoting innovation and growth.鈥� Saltelli says that, on average, two million articles are published in 30,000 journals each year. This 鈥渕onster鈥� cannot obviously not be controlled by the same gentlemen鈥檚 agreements, and with the same ethos as when everyone knew everyone, he believes.聽

鈥淲hen there are few players and they know each other, you can have behaviour that is more constructive than in the virtually Hobbesian situation we are now in where there are millions of us, and everyone needs to make a living and promote their careers.鈥澛�

Do not read this! 鈥淲here鈥檚 the crisis? There is no crisis. Science is a never-ending process.鈥澛�

Daniele Fanelli is a researcher at the London School of Economics and himself a prominent figure within 鈥渕eta-science,鈥� i.e. research on research.聽

鈥淚t鈥檚 ironic that part of the core evidence basis for that there is indeed a reproduction crisis, if you take a dispassionate look at it, does not show this to be the case at all,鈥� says Fanelli via Skype.聽

He would like to properly lay out his views on the so-called crisis: 鈥淗ow much time do you have?鈥� he asks.聽

Fanelli has written one of several articles in the journal PNAS which now warns against newspaper articles such as this one. After the difficulties with reproducing research results in psychology and medicine became known, articles about science that is in crisis, or even聽鈥渂roken,鈥� have appeared multiples times both in scientific literature and specialist science reporting. Stories about fraud and immorality, erroneous, stupid and unreliable research are captivating reading material. But critics believe the crisis-maximisation undermines confidence in science and makes it easier for those who, for ideological reasons, would like to discredit research areas, for example in the case of vaccines, genetic modification or climate change.聽

Fanelli reviews the knowledge base for the crisis claims in an article. When you look closely, you can鈥檛 find support for the dramatic conclusion, he believes. To the contrary. Referring to several new and successful attempts to repeat earlier research, Fanelli points out studies that聽indicate that unethical or dubious research practices are not more common than before. For example, the spike in the number of articles withdrawn is probably mainly due to the fact that the journals have become more diligent. While it is true that you do see some false-positive findings and p-hacking, Fanelli believes this has little impact when you compile and summarise the research in an area.聽

Fanelli has made a career of studying cheating in research, questionable practices and reliability in research. He says that one of the reasons he 鈥渆nthusiastically鈥� entered this discipline was that he had 鈥渁 feeling鈥� that something had to be dysfunctional.聽

鈥淚f I had seen a knowledge base that supports that viewpoint, I would have supported it. It鈥檚 not that I have an agenda. I just think it鈥檚 wrong.鈥澛�

So you have changed your mind?聽

鈥淎bsolutely, although I don鈥檛 think I ever talked about a 鈥榗risis.鈥欌��

Positive psychology. Much psychological research is simply 鈥減sychobabble,鈥� concluded the newspaper The Independent following the hitherto biggest attempt to review psychological research in 鈥淭he Reproducibility Project鈥� in 2015. 鈥淥nly鈥� 40 of the 100 experiments were successfully recreated on a large scale. Fanelli looks at the study as a turning point, but with opposite signs. Before this, you could easily get the impression that psychology did not have any standards for presenting evidence.聽

- 鈥淚 assumed that practically no research in psychology could be reproduced. Not because it鈥檚 that easy to be a fraudster in this area, but because humans are so complex that you can鈥檛 expect it to work the same way as in particle physics. If there鈥檚 one field that鈥檚 really in trouble, it鈥檚 psychology. But for me, 40 per cent was pretty good; you can鈥檛 logically expect 100 per cent. And this was supposed to be the discipline in the worst shape! And then the statisticians started looking at the numbers, and all the reanalyses suggest that the estimates really are better,鈥� says Fanelli.

Do you fear that the narrative is being hijacked by people with a special agenda in the debate on research policy and the use of research in politics?聽

鈥淚 saw people use my own studies to support scepticism against climate science, or people who said: 鈥榃hy should we vaccinate our children if biomedical research is corrupt and broken,鈥欌�� says Fanelli.聽

鈥淭his is a zeitgeist that questions all types of expertise. This is therefore a challenge that must be met.聽

Fanelli is an avid proponent of better training in statistics and other initiatives that can make research more robust. However, he has little belief in the notion that bad or unreliable research is due to the pressure to publish. People were complaining about the pressure to publish as early as the 1950s and certainly before then too, he points out. And he refers to his own research, which shows that scientists simply do not publish more than they did before if you take into account the fact that articles today frequently have several co-authors. Some surveys also suggest that published studies are getting longer, more sophisticated, and richer in data content.聽

鈥淚n many ways it runs counter to what the classic pressure to publish tale preaches.鈥� why are people so concerned about the dangers of publishing pressure?聽

鈥淚 suspect it鈥檚 a narrative that serves the academics themselves. No one wants to be pressured,鈥� says Fanelli.聽

Defence. 鈥淪cience can鈥檛 be self-correcting as long as the incentive schemes produce poor science,鈥� says Saltelli, who has read Fanelli鈥檚 article in PNAS.聽

He says the amount of bad research far exceeds the number of withdrawn articles, and points out that the journals still do not encourage enough replication studies. It鈥檚 still the case that these studies mainly appear in print when scientists do not manage to confirm the research. This proves, he believes, that the system is still not equipped to handle the 鈥渃oming crisis.鈥澛�

In a currently unpublished response to the crisis critics in PNAS, Saltelli, together with 幸运飞艇计划聽colleague Silvio Funtowicz, one of the founders of the 鈥減ost-normal鈥� way of thinking, write that they think Fanelli鈥檚聽criticism resembles a 鈥渞eligious view,鈥� complete with the idea聽that 鈥渄oubters are corrupting young people.鈥� But what about his own approach. What does he聽think about it potentially being used, for example, to justify opposition to vaccines or deny climate change?聽

Congressional battle. It was apparently an ordinary presentation of a report on Capitol Hill in Washington, D.C. Called 鈥淭he Irreproducibility Crisis of Modern Science,鈥� the report was received with open arms on 17 April this year by Texas Republican Lamar Smith, chair of the House Committee on Science, Space and Technology.聽

This was "an important study," said Smith, according to Undark Magazine. The report could give the impression of being a serious and neutral overview of the 鈥渃risis鈥� in science, with several well-known proposals for how research could be improved.聽

It was published by the National Association of Scholars, a conservative research policy group best known for its support of climate change sceptics. In an article in the Wall Street Journal, the authors of the report elaborated on their views: 鈥淭he whole climate science discipline is a jumble of unreliable statistics, random research methods and politicised groupthink,鈥� they wrote.聽

The debate about reproducibility is already being exploited by political activists, declared Naomi Oreskes, historian of science at Harvard and known for her book Merchants of Doubts, to Undark: 鈥淐limate sceptics and other activists are having a feeding frenzy because this is exactly what they want. And what they want to do is use this now to try to discredit all science.鈥澛�

Among other things, the report supports a bill that would bar the US Environmental Protection Agency (EPA) from using research that is not 鈥渟ubstantially reproducible.鈥澛�

This could prevent the environmental authorities from taking advantage of the best knowledge in the regulation of, for example, environmental toxins because of lingering scientific uncertainty or because data is kept secret, for example, for privacy reasons. Scandal-ridden EPA Administrator Scott Pruitt, who was appointed by Donald Trump, is now trying to introduce this rule, over the strong protests of American research organisations. His proposal, which is now being circulated for comment, points directly to the 鈥渞eplication crisis鈥� as justification.聽

They are indeed the 鈥渆nemies of science,鈥� admits Saltelli about the authors behind the report submitted to Congress.聽

鈥淭hey are turning the knowledge base into a weapon, there is no easier way to put it,鈥� he writes in an e-mail.聽

The Italian fears that events like this will cause scientists to聽not speak openly about issues in science. However, as he has also written, he believes that the combination of corruption, anger and new digital technology can mobilise change: Because science needs a reformation.聽

jkt@morgenbladet.no