[e2e] was double blind, now reproduceable results

[David P. Reed]

At 07:56 PM 5/19/2004, Bob Braden wrote:
>This is often hard to do, and I cannot deny that in many cases we are
>mightily challenged to come up with an appropriate measure X.  However,
>I would suggest that this is the ideal towards which we should strive.

Reproducing results is not always the appropriate goal.   When it is, I 
agree wholeheartedly.   When it cannot be, there are other measures of 
experimental and empirical quality.   Sharing one's tools and techniques, 
methods and materials, data collected, etc. are equally important.

>We should not just say, "It's too hard to do."

I have the impression you are trying to claim that there are only two 
approaches, one hard and one lazy.   Nothing I have said implies that we 
should just say "it's too hard".   My point is that the work of scientific 
review is hard, and frivolous attempts to reduce it to a simplistic 
cookbook rule such as "all experiments must be reproducible before we allow 
them to be reported in print or in conferences" would have ruled out 90% of 
the scientific breakthrough experiments I most admire.   I would resist 
such a rule, and though I share Jon's disappointment with the quality of 
many papers in our field, this idea that there is a simple way to solve the 
reviewer's task could be characterized as lazy.

The job of producing a refereed paper involves hard work on both 
sides.  When a paper does not provide enough information to judge its 
contribution, that's the reviewer's job to point out.   If there are good 
reasons why the paper should be published, even if the results may not be 
reproducible, the job of science is to come to terms with those reasons, 
discuss better methods, suggest other tests, etc.   But suppressing 
publication based on mere matters of conformance to a mythical pure 
Scientific Method seems absurd and extreme.

Science is a self-correcting process, and what we must call for is a much 
harder standard than the trivial one, which is the standard of intellectual 
integrity.   That is my creed, and that is not laziness.