A summary of the evidence that most published research is false | Simply Statistics

A summary of the evidence that most published research is false | Simply Statistics

A summary of the evidence that most published research is false

Posted on December 16, 2013 by Jeff Leek

One of the hottest topics in science has two main conclusions:

• Most published research is false
• There is a reproducibility crisis in science

The first claim is often stated in a slightly different way: that most results of scientific experiments do not replicate. I recently got caught up in this debate and I frequently get asked about it.
So I thought I'd do a very brief review of the reported evidence for the two perceived crises. An important point is all of the scientists below have made the best effort they can to tackle a fairly complicated problem and this is early days in the study of science-wise false discovery rates. But the take home message is that there is currently no definitive evidence one way or another about whether most results are false.

1. Paper: Why most published research findings are false. Main idea: People use hypothesis testing to determine if specific scientific discoveries are significant. This significance calculation is used as a screening mechanism in the scientific literature. Under assumptions about the way people perform these tests and report them it is possible to construct a universe where most published findings are false positive results. Important drawback: The paper contains no real data, it is purely based on conjecture and simulation.
2. Paper: Drug development: Raise standards for preclinical research. Main idea: Many drugs fail when they move through the development process. Amgen scientists tried to replicate 53 high-profile basic research findings in cancer and could only replicate 6. Important drawback: This is not a scientific paper. The study design, replication attempts, selected studies, and the statistical methods to define "replicate" are not defined. No data is available or provided.
3. Paper: An estimate of the science-wise false discovery rate and application to the top medical literature. Main idea: The paper collects P-values from published abstracts of papers in the medical literature and uses a statistical method to estimate the false discovery rate proposed in paper 1 above. Important drawback: The paper only collected data from major medical journals and the abstracts. P-values can be manipulated in many ways that could call into question the statistical results in the paper.
4. Paper: Revised standards for statistical evidence. Main idea: The P-value cutoff of 0.05 is used by many journals to determine statistical significance. This paper proposes an alternative method for screening hypotheses based on Bayes factors. Important drawback: The paper is a theoretical and philosophical argument for simple hypothesis tests. The data analysis recalculates Bayes factors for reported t-statistics and plots the Bayes factor versus the t-test then makes an argument for why one is better than the other.
5. Paper: Contradicted and initially stronger effects in highly cited research Main idea: This paper looks at studies that attempted to answer the same scientific question where the second study had a larger sample size or more robust (e.g. randomized trial) study design. Some effects reported in the second study do not match the results exactly from the first. Important drawback: The title does not match the results. 16% of studies were contradicted (meaning effect in a different direction). 16% reported smaller effect size, 44% were replicated and 24% were unchallenged. So 44% + 24% + 16% = 86% were not contradicted. Lack of replication is also not proof of error.
6. Paper: Modeling the effects of subjective and objective decision making in scientific peer review. Main idea: This paper considers a theoretical model for how referees of scientific papers may behave socially. They use simulations to point out how an effect called "herding" (basically peer-mimicking) may lead to biases in the review process. Important drawback: The model makes major simplifying assumptions about human behavior and supports these conclusions entirely with simulation. No data is presented.
7. Paper: Repeatability of published microarray gene expression analyses. Main idea: This paper attempts to collect the data used in published papers and to repeat one randomly selected analysis from the paper. For many of the papers the data was either not available or available in a format that made it difficult/impossible to repeat the analysis performed in the original paper. The types of software used were also not clear. Important drawback: This paper was written about 18 data sets in 2005-2006. This is both early in the era of reproducibility and not comprehensive in any way. This says nothing about the rate of false discoveries in the medical literature but does speak to the reproducibility of genomics experiments 10 years ago.
8. Paper: Investigating variation in replicability: The "Many Labs" replication project. (not yet published) Main idea: The idea is to take a bunch of published high-profile results and try to get multiple labs to replicate the results. They successfully replicated 10 out of 13 results and the distribution of results you see is about what you'd expect (see embedded figure below). Important drawback: The paper isn't published yet and it only covers 13 experiments. That being said, this is by far the strongest, most comprehensive, and most reproducible analysis of replication among all the papers surveyed here.

I do think that the reviewed papers are important contributions because they draw attention to real concerns about the modern scientific process. Namely

• We need more statistical literacy
• We need more computational literacy
• We need to require code be published
• We need mechanisms of peer review that deal with code
• We need a culture that doesn't use reproducibility as a weapon
• We need increased transparency in review and evaluation of papers

Some of these have simple fixes (more statistics courses, publishing code) some are much, much harder (changing publication/review culture).
The Many Labs project (Paper 8) points out that statistical research is proceeding in a fairly reasonable fashion. Some effects are overestimated in individual studies, some are underestimated, and some are just about right. Regardless, no single study should stand alone as the last word about an important scientific issue. It obviously won't be possible to replicate every study as intensely as those in the Many Labs project, but this is a reassuring piece of evidence that things aren't as bad as some paper titles and headlines may make it seem.

Many labs data. Blue x's are original effect sizes. Other dots are effect sizes from replication experiments (http://rolfzwaan.blogspot.com/2013/11/what-can-we-learn-from-many-labs.html)

The Many Labs results suggest that the hype about the failures of science are, at the very least, premature. I think an equally important idea is that science has pretty much always worked with some number of false positive and irreplicable studies. This was beautifully described by Jared Horvath in this blog post from the Economist.  I think the take home message is that regardless of the rate of false discoveries, the scientific process has led to amazing and life-altering discoveries.

FDA examining antibacterial soaps, body washes - CNN.com

FDA examining antibacterial soaps, body washes - CNN.com

(CNN) -- Manufacturers of antibacterial hand soap and body wash will be required to prove their products are more effective than plain soap and water in preventing illness and the spread of infection, under a proposed rule announced Monday by the Food and Drug Administration.

Those manufacturers also will be required to prove their products are safe for long-term use, the agency said.

 

"Millions of Americans use antibacterial hand soap and body wash products," the agency said in a statement. "Although consumers generally view these products as effective tools to help prevent the spread of germs, there is currently no evidence that they are any more effective at preventing illness than washing with plain soap and water.

 

"Further, some data suggest that long-term exposure to certain active ingredients used in antibacterial products -- for example, triclosan (liquid soaps) and triclocarban (bar soaps) -- could pose health risks, such as bacterial resistance or hormonal effects."

About 2,000 individual products contain these products, health officials said.

 

"Our goal is, if a company is making a claim that something is antibacterial and in this case promoting the concept that consumers who use these products can prevent the spread of germs, then there ought to be data behind that," said Dr. Sandra Kweder, deputy director of the Office of New Drugs in FDA's Center for Drug Evaluation and Research.

 

"We think that companies ought to have data before they make these claims."

Studies in rats have shown a decrease in thyroid hormones with long-term exposure, she said. Collecting data from humans is "very difficult" because the studies look at a long time period.

 

Get dangerous germs out of your home

 

Before the proposed rule is finalized, companies will need to provide data to support their claims, or -- if they do not -- the products will need to be reformulated or relabeled to remain on the market.

"This is a good first step toward getting unsafe triclosan off the market," said Mae Wu, an attorney for the Natural Resources Defense Council. "FDA is finally taking concerns about triclosan seriously. Washing your hands with soap containing triclosan doesn't make them cleaner than using regular soap and water and can carry potential health risks.

 

The FDA first proposed removing triclosan from certain products in 1978, the council said, "but because the agency took no final action, triclosan has been found in more and more soaps."

In 2010, the council said it sued FDA to force it to issue a final rule. The new proposed rule stems from a settlement in that suit, according to the NRDC.

 

The rule is available for public comment for 180 days, with a concurrent one-year period for companies to submit new data and information, followed by a 60-day period for rebuttal comments, according to the FDA.

 

The target deadline is June 2014 for the public comment period, then companies will have until December 2014 to submit data and studies. The FDA wants to finalize the rule and determine whether these products are "generally recognized as safe and effective" by September 2016.

"Antibacterial soaps and body washes are used widely and frequently by consumers in everyday home, work, school and public settings, where the risk of infection is relatively low," said Dr. Janet Woodcock, director of the FDA's Center for Drug Evaluation and Research.

 

"Due to consumers' extensive exposure to the ingredients in antibacterial soaps, we believe there should be a clearly demonstrated benefit from using antibacterial soap to balance any potential risk."

The action is part of FDA's ongoing review of antibacterial active ingredients, the agency said.

Hand sanitizers, wipes and antibacterial products used in health care settings are not affected.

Most hand sanitizers have 60% alcohol or ethanol and are generally recognized as safe when water isn't available, Kweder said. However, health officials still believe washing hands with soap and water is the best method.

Cassini reveals clues about Titan’s hydrocarbon lakes and seas | Science Recorder

Cassini reveals clues about Titan’s hydrocarbon lakes and seas | Science Recorder

P.J. O’Rourke - American Satirist, Journalist and Author | Point of Inquiry

P.J. O’Rourke - American Satirist, Journalist and Author | Point of Inquiry

The News About the Universe Isn't Good

The News About the Universe Isn't Good

Hitchens didn't think rejecting religion would solve everything. But he knew only reason would give us justice by Jeffrey Tayler @ Salon

Christopher Hitchens (Credit: AP/Chad Rachman)

 

David Strumfels -- You might prefer the original article at

http://www.salon.com/2013/12/14/the_real_new_atheism_rejecting_religion_for_a_just_world/

 

The ever-polemical atheist author Christopher Hitchens died two years ago this month, yet his incisive, erudite diatribes against religion continue to rile the faithful and spark debate. The latest anti-Hitch outburst comes from Sean McElwee, a writer and researcher of public policy who describes himself as “a poorly practicing Christian who reads enough science to be functional at dinner parties.” McElwee calls for a “truce” between believers and nonbelievers. But he stands on the losing side of both public opinion trends and history. According to a Pew poll conducted in 2012, a record number of young Americans – a quarter of those between the ages of 18 and 29 — see themselves as unaffiliated with any religion. Atheists’ ranks are swelling, and believers are finding it increasingly difficult to justify their faith.

McElwee begins by calling the New Atheist movement “a rather disturbing trend” in a country “whose greatest reformer” – Martin Luther King, Jr. – “was a Reverend.” Dr. King won fame as a civil rights leader, not as a religious figure. McElwee would do well to recall the words of Founding Father John Adams: “the Government of the United States of America is not, in any sense, founded on the Christian religion.” McElwee goes on to attribute to New Atheists an unsound premise of his own concoction:

1.  The cause of all human suffering is irrationality
2.  Religion is irrational
3.  Religion is the cause of all human suffering

Hitchens’ most notorious atheistic tome is entitled “God Is Not Great: How Religion Poisons Everything.” But no serious reader could conclude from this book (or from the writings of the other New Atheists — Richard Dawkins, Sam Harris and Daniel Dennett — whom McElwee also hopes to debunk) that he considers religion the sole wellspring of humankind’s woes. Though he derided religion long before and after he published “God Is Not Great,” Hitchens never said any such thing, and no reasonable person would believe it. Are cancer and flesh-eating bacteria manifestations of irrationality? What about about wars over territory or natural resources? Poverty and inequality?
Bullying and bulimia? The “classical logical error of post hoc ergo propter hoc” McElwee ascribes to New Atheists simply does not exist.

McElwee then jumps to Hitchens’ (misbegotten) support of the second Iraq war and attempts to press it into service to discredit him in matters of faith. Hitchens, as McElwee correctly notes, opposed the 1991 invasion of Iraq, but when George W. Bush was in office, according to McElwee, Hitchens “decided that, in fact, bombing children was no longer so abhorrent” because the 2003-2011 conflict was to be a “final Armageddon between the forces of rationality and the forces of religion.” No, this was not how Hitchens viewed the second Iraq war. He advocated invading Iraq to overthrow Saddam, who was, he contended, guilty of crimes against humanity, and he (mistakenly) assumed a stable democracy would result from the dictator’s ouster.

Hitchens understood the secular nature of Saddam’s Ba’ath Party, which made all the more puzzling and problematic his stubborn insistence that Saddam was colluding with Al Qaeda. But McElwee then asserts that “the force of rationality and civilization was led by a cabal of religious extremists” – in the Bush administration — which “was of no concern for Hitchens.” George W. Bush was a convert to Evangelical Christianity, which does not necessarily make him a “religious extremist,” and the (mixed) faiths of the Iraq War’s other architects (Dick Cheney, Condoleeza Rice, Paul Wolfowitz, Douglas Feith, et al.) did not fuel their zeal for deposing Saddam.

McElwee proceeds to mischaracterize Hitchens’ post-9/11 worldview as a “war between the good Christian West and the evil Muslim Middle East.” How McElwee can expect us to believe this of Hitchens, who authored a book (“The Missionary Position”) denouncing Mother Theresa as a fraud and relentlessly attacked Christianity, baffles me, as does McElwee’s blindness to his own blunder. Is Hitchens now, according to him, pro-Christian?

McElwee also falsely attributes obscurantist motives to New Atheists. “Might it be better to see jihad as a response to Western colonialism and the upending of Islamic society, rather than the product of religious extremism? The goal of the ‘New Atheists’ is to eliminate centuries of history that Europeans are happy to erase, and render the current conflict as one of reason versus faith rather than what is, exploiter and exploited.”

Stripping jihad of its religious grounds invites nothing but confusion. Jihad in Arabic means “struggle,” but, with respect to Islam, denotes “a struggle in the name of faith,” which includes holy war against infidels waged as a matter of religious duty.  Such jihad is, ipso facto, religious. Informed readers also know that jihadists, in their addresses to the Muslim umma, rail against Western occupation of Islamic lands, “infidel” Western-backed dictators in Muslim countries, and so on — all the while citing passages from the Quran. Hitchens and Dawkins, both Europe-born and versed in their continent’s past – a past replete with religious and political conflicts of all kinds — have never sought to “erase” its history or present “the current conflict” as solely one of “reason versus faith.”
McElwee then tendentiously defines religion so as to paper over its often decisive role in precipitating conflicts. Though he allows that it might “motivate acts of social justice and injustice,” “[r]eligion is both a personal search for truth as well as a communal attempt to discern where we fit in the order of things.” Religion first and foremost consists of unsubstantiated, dogmatically advanced explanations for the cosmos and our place in it, with resulting universally applicable rules of conduct. A good many of these rules – especially those regarding women’s behavior and their (subservient) status vis-à-vis men, and prescriptions for less-than-merciful treatment of gays – are repugnant, retrograde, and arbitrary, based on “sacred texts” espousing “revealed truths” dating back to what the British atheist philosopher Bertrand Russell justly called the “savage ages.” (Islam by no means has a monopoly on such rules – check Leviticus for its catalogue of “crimes”: working on the Sabbath, cursing one’s parents, being the victim of rape – that merit the death penalty.) Just how such “holy” compendia of ahistorical, often macabre fables are supposed to help anyone in a “personal search for truth” mystifies me.

Lacking any alternative, McElwee then tells nonbelievers to lay off the faithful: “any critique of religion that can be made from the outside (by atheists) can be made more persuasively from within religion.” The last time I checked, those “within religion” who denounce religion as untrue, unfounded on fact, irrational by its nature and preying upon our fears, would in fact be atheists. The problem is not, as McElwee says, “the Church’s excesses” – but the Church itself, its backward rules, its reactionary ethos, its groundless assertion of moral authority. The latter is laughable, especially regarding the Catholic Church, in view of the catalogue of crimes – including the persecution of Jews, the Crusades, the Inquisition and silence with respect to Hitler’s Final Solution — for which it bears self-admitted guilt. If one breaks free of the racket of faith, then faith-sanctioned strictures, fantastic tales (human parthenogenesis among them) demanding faith to be believed, to say nothing of the justness of tax exemptions for faith organizations, all appear as entirely human creations that are questionable at best, criminal at worst, and certainly deserving of no kid-glove treatment.

“The impulse to destroy religion will ultimately fail,” McElwee claims. Just what he means by this is unclear. Hitchens spoke out tirelessly against religion but never believed it could be eradicated; rather, he likened it to Camus’ plague-infected rats, scurrying about in humanity’s sewer, ever awaiting a chance to reemerge. Hitchens certainly never foresaw the bizarre scenario McElwee outlines: “Banish Christ and Muhammad and you may end up with religions surrounding the works of Zizek and Sloterdijk (there is already a Journal of Zizek Studies, maybe soon a seminary?).
Humans will always try to find meaning and purpose in their lives, and science will never be able to tell them what it is.” New Atheists have never assigned science such a role. Hitchens himself often recommended the consolations of literature for this purpose. The broader point rationalists make is simple: People, having set aside fairy tales and mandated moral certainties delivered from on high, must seek meaning on their own, seek to order society in ways beneficial for all, and do so with reason as lodestar.

So what is to be done? McElwee trots out the idea of a truce – “one originally proposed by the Catholic church and promoted by the eminent Stephen J. Gould,” that “Science, the study of the natural world, and religion, the inquiry into the meaning of life (or metaphysics, more broadly) constitute non-overlapping magisteria.” One straightaway must regard as suspect a “truce” advocated by an organization guilty of repressing scientists and opposing the scientific Weltanschauung. And one would be right to be suspicious, according to McElwee’s proposition: “Neither [science nor religion] can invalidate the theories of the other, if such theories are properly within their realm.” Just what the boundaries of those realms are and who decides them have been matters of contention since time immemorial. Just ask Galileo.

McElwee next concludes that “religion (either secular or theological) does not poison all of society and science should not be feared, but rather embraced.”

No one is waiting for McElwee’s green light to “embrace” science, which holds its place among us by virtue of its proven utility, its lab-tested veracity. At this point, McElwee’s second citation of Martin Luther King cannot avail him. “Science deals mainly with facts; religion deals mainly with values. The two are not rivals.” Dr. King’s saying this does not make it so. Faith and reason are fighting for supremacy the world over, and rationalists must make their case with ardor, shying away from no battle. Atheists who wobble in defense of nonbelief would do well to recall 9/11, Baruch Goldstein’s Hebron massacre of Palestinians, the Salem witch trials and violence meted out in the name of religion to “unchaste” women throughout the ages. This is, of course, an incomplete list of atrocities motivated by religion.

The sooner we accord priests, rabbis and imams the same respect we owe fabulists and self-help gurus, the faster we will progress toward a more just, more humane future. Enlightenment must be our goal, and that was what Hitchens advocated above all.

 

                

Jeffrey Tayler is a contributing editor at The Atlantic. His seventh book, "Topless Jihadis -- Inside Femen, the World's Most Provocative Activist Group," will be published on December 20 as an Atlantic ebook. Follow @JeffreyTayler1 on Twitter. More Jeffrey Tayler.