This article from the Economist details how an amateur palaeontologist uncovered serious errors in previously published papers on how dinosaurs grew.
In total, he examined 12 papers. In four cases the original data had been lost. In three, the statistics were correct. Three had what were, in his opinion, serious errors that invalidated the principal conclusions made about the growth of seven of the 11 dinosaur species those papers looked at. (He found, for example, that Tyrannosaurus’s peak growth rate had been overstated by a factor of two.) And two papers were reviews that used data from these three, and were thus also in error.
Many of his observations seemed, to him, particularly odd in light of the graphs in the original papers, so he looked at these, too. When he did so he discovered that some of the graphs’ points did not correspond to the data their captions referred to. In one case, involving a genus called Massospondylus (ancestral members of the group that includes animals like Diplodocus), he found that only four of nine points on the graph came from data cited in the paper.
So, kudos to Dr Nathan Myhrvold for having the wherewithal to challenge the orthodoxy! Unfortunately, as we see in the unfolding debate about the role of big data in the confirmation or refutation of economic models, the refutees are having a hard time swallowing that they were wrong with the fallacy of appealing to authority:
We understand that Myhrvold has questioned our methodology. In all instances our methods and results were subject to professional peer review, and our findings have been supported time and time again by others. Myhrvold’s current reinterpretation of our data, although reaching moderately different conclusions on a species by species basis, strongly supports the cardinal conclusions that we reached regarding how dinosaurs grew.
The numbers were off, sometimes by a lot, but fortunately our basic idea that dinosaurs grew quickly is not being overturned. That would have been a disaster since it is so well accepted.
The current peer review system is broken, but even that is meaningless if scientists are unwilling to change their understanding of the world in the light of new data.