Information on possible true effect sizes that are compatible with the data must be obtained from the point estimate, e.g., from a sample average, and from the interval estimate, such as a confidence interval. Also larger p-values offer some evidence against the null hypothesis, and they cannot be interpreted as supporting the null hypothesis, falsely concluding that ‘there is no effect’. Consistent with the recommendations of the late Ronald Fisher, p-values should be interpreted as graded measures of the strength of evidence against the null hypothesis. Data dredging, p-hacking, and publication bias should be addressed by removing fixed significance thresholds. But current incentives to hunt for significance lead to selective reporting and to publication bias against nonsignificant findings. Interpreting inflated significant results while ignoring nonsignificant results will thus lead to wrong conclusions. One reason is that with anything but ideal statistical power, significant effect sizes will be biased upwards.
However, applying significance thresholds makes cumulative knowledge unreliable. Reliable conclusions on replicability and practical importance of a finding can only be drawn using cumulative evidence from multiple independent studies. Many apparent replication failures may thus reflect faulty judgment based on significance thresholds rather than a crisis of unreplicable research. A replication can therefore not be interpreted as having failed only because it is nonsignificant. Also significance ( p ≤ 0.05) is hardly replicable: at a good statistical power of 80%, two studies will be ‘conflicting’, meaning that one is significant and the other is not, in one third of the cases if there is a true effect. In either case, p-values tell little about reliability of research, because they are hardly replicable even if an alternative hypothesis is true. A major problem is that we tend to take small p-values at face value, but mistrust results with larger p-values.
We review why degrading p-values into ‘significant’ and ‘nonsignificant’ contributes to making studies irreproducible, or to making them seem irreproducible. The widespread use of ‘statistical significance’ as a license for making a claim of a scientific finding leads to considerable distortion of the scientific process (according to the American Statistical Association).
0 kommentar(er)
