[e2e] was double blind, now reproduceable results

[David G. Andersen]

On Thu, May 20, 2004 at 09:23:02AM -0500, Saad Biaz scribed:
> 
> Reproducible results, checking, .... It sounds to me that academic
> integrity is dead. Worse, it seems that we accept that it is dead.
> Now, we want to set a batterie of checks... This is Tom Ridge's approach,
> forgetting the source of the problem...

Most reasons to attempt to reproduce earlier results have nothing
to do with integrity:

  * Unstated assumptions
  * Refining or correcting hypotheses
  * Bugs/etc.  (How much code do we all write w/in 24 hours of the deadline?)
  * Measurement error
  * Re-testing previous results in a somewhat different environment

One of the really nice things about being forced to make your experiments
as reproducible as possible is that you have to go through and examine
all of your assumptions and carefully describe your experimental process.
This is probably as revealing to the authors as it is to the readers
in terms of understanding what it is you've actually done.

  "The most exciting phrase to hear in science, the one that heralds
   new discoveries, is not 'Eureka!' (I found it!) but 'That's funny...'"
     --Isaac Asimov

(thanks to Mike Dahlin's page for that one).  Finding results that 
contradict earlier results is an opportunity to dig down and find
a better answer.
 
> of thought, and ..... particularly because we hated becoming a
> salesperson. Here we are that now where many researchers must behave
> like a salesperson using hype, deceit, unreproducible results....

There are undoubtedly people in any community who will fudge their
results in some way (witness very high profile examples with discovery
of new chemical elements).  But this is not due to pressure to
publish happy results - it's because they were able to obtain {fame,
funding, tenure, etc.} by doing it.  It's an unavoidable problem
in any circumstances.  Does computer science / networking have more
of a problem with intentional dishonesty than any other field?

Being a salesperson is the reality for all scientists.  Why give a
good conference talk?  Why announce your cool new system on
e2e-interest?  Why make your grad students read Strunk & White and
spend hours editing their papers?  Because you're justifiably proud
of your work and want to make sure it has impact.  Don't confound
sales and dishonesty.  We should encourage one (to the proper degree!)
and strongly, strongly discourage the other.

  -Dave

-- 
work: dga at lcs.mit.edu                          me:  dga at pobox.com
      MIT Laboratory for Computer Science           http://www.angio.net/