[e2e] was double blind, now reproduceable results

[RJ Atkinson]

On May 19, 2004, at 09:52, Jon Crowcroft wrote:
> also we should stop accepting papers for conferences and
> workshops and journals that report work done on linux or
> ns or any other open source platform where the
> imlementation is not available for others to test the
> work - we should also _DESIGNATE_ a certain percentage of
> slots in these events for the sole purpose of publishjing
> results that reproduce existing work (or not) as other
> subjects (medicine) do...

I am a big fan of experimental computer science.  Part
of experimental science is having independent 3rd parties
re-run the experiments to validate the reproducibility
of results.

> we quite liked
> http://www.clarkson.edu/class/cs644/xen/whyrepeat.html
> for example - but if it had not been the same, we ought
> still to have liked it:)
>
> so if we used online discussion first, this could
> pre-filter stuff (people could check stuff before a paper
> went in, then get acked (or added as co-authors if they
> help improve or fix irreproduceable results)
>
> I am not saying publishin the experimental instruments
> will guarantee correctness, but it will improve quality,
> and it will let us eliminate misunderstandings (viz
> the Cold Fusion of the Internet:-)

Agree.

	Not everything will fall into the category where
the source ought to be made available (e.g. it is unlikely
any commercial firm would release proprietary software),
but there ought to be a fair number of instances (e.g. GPL
source, some non-profit/academic work) where it is entirely
practical.  For example, many research organisations routinely
do release some material and probably could (if they chose)
release enough of their material (including the experimental
process steps in detail) on an http/ftp site so that 3rd parties
can re-run the experiment.

	By the way, I'm also a fan of laboratory-based undergraduate
computer science instruction for similar reasons.  In some
cases, really good research experiments might later become
teaching lab experiments for students (as they are long since
in Chemistry or Physics).

Cheers,

Ran
rja at extremenetworks.com