[e2e] Is sanity in NS2?

[RJ Atkinson]

On Sep 14, 2005, at 15:58, Mark Allman wrote:
> Nobody said it was perfect.  But, to assert that there is nothing done
> in this area of validation or regression or quality control or  
> whatever
> you want to call it is just wrong, IMHO.  That is all I was trying to
> say.

I think a large part of the issue is that many of the "externally
contributed" NS models have had little or no credible validation,
unlike the solid models done by some others (e.g. Floyd).  This is
then compounded by those ns users who don't validate their particular
simulation scenario before believing its results.

Unfortunately, submitted papers that both use unvalidated models
and use them in an unvalidated scenario are all too common.  Even
more scary to me, some authors who submit papers to IEEE periodicals
don't seem to understand what validation means when the reviewer
asks them to edit the paper to discuss how the simulation model
and scenario were validated so the reader can have some basis
for believing the paper.

> Researchers are responsible for ensuring that their experiments
> are sound.  Doesn't matter if you're using opnet or ns2 or two
> Solaris boxes and a Cisco router or a pencil and paper.

Quite so.  Unfortunately, this reviewer sees a lot more poorly
designed simulations being written up and submitted to periodicals
than poorly designed laboratory experiments (or, for that matter,
papers with conclusions based on incorrect maths).

I think the root problem here is that many universities
are encouraging students to run simulations without teaching
students about how to use simulation properly and without
sufficient supervision that the simulation was used properly.
This is unfortunate.

I'm not sure what is available in the line of standard textbooks,
but if one doesn't yet exist maybe there would be room for someone
to write up a textbook on when and how to simulate communications
systems -- or maybe even a text on good experimental design
more generally.

Yours,

Ran