[e2e] Is sanity in NS2?

[Detlef Bosau]

RJ Atkinson wrote:
> 
> On Sep 14, 2005, at 15:58, Mark Allman wrote:
> 
>> Nobody said it was perfect.  But, to assert that there is nothing done
>> in this area of validation or regression or quality control or  whatever
>> you want to call it is just wrong, IMHO.  That is all I was trying to
>> say.
> 
> 
> I think a large part of the issue is that many of the "externally
> contributed" NS models have had little or no credible validation,
> unlike the solid models done by some others (e.g. Floyd).  This is


This what I wanted to call "core" yesterday: The stable part (perhaps 
the _major_ part) in the NS in contrast to many hacks which are not 
validated and even not known to the public. E.g. my PTE stuff. Nobody in 
the community ever read it. No one can say whether results using this 
code are certain or not.

And one remark to the validation runs in the NS. As far as I see, their 
purpose ist not primarily to validate the modules but the compilation. 
It is validated whether the compiled and installed NS yields the 
expected results for a huge number of tests.


> then compounded by those ns users who don't validate their particular
> simulation scenario before believing its results.
> 
> Unfortunately, submitted papers that both use unvalidated models
> and use them in an unvalidated scenario are all too common.  Even

And, again refering to my own work, it´s hardly possible to do this in 
some other way.

Of course we could discuss whether I first write my code, then ask 
someone (btw: whom?) whether I may contribute the code, then wait for 
some people to approve it and then, when I get older, loosing my hair, 
many years from know (The Beatles) I perhaps get the chance to have a 
paper published.

> more scary to me, some authors who submit papers to IEEE periodicals
> don't seem to understand what validation means when the reviewer
> asks them to edit the paper to discuss how the simulation model
> and scenario were validated so the reader can have some basis
> for believing the paper.
> 

And this is something you have to learn and where you need assistance. 
That´s why I complain that nobody teached me there. I know this is 
annoying to many of you. However, it is somewhat unfortunate to do this 
all from scratch and the only feedback you have is the one rejected 
paper every year. Personally, I have only a rough idea, how I could 
validate my own code. I can describe the PTE algorithm. I can run a 
number of test scenarios. I can compare the output in the logfile with 
the intended behaviour of the algorithm. So, I could say: Every scenario 
yields the correct result. But is this sufficient?

Unfortunately, the typical attitude is that: If the paper is accepted, 
your teacher was correct in his orders he has given to you and you 
followed them corectly. If the paper was rejected, your teacher was 
.....to you and you did not follow them correctly.

I have no teacher, so the former does not apply.

Hoewever: I still do not fully undertand the standards, by which a paper 
is accepted or rejected.

>> Researchers are responsible for ensuring that their experiments
>> are sound.  Doesn't matter if you're using opnet or ns2 or two
>> Solaris boxes and a Cisco router or a pencil and paper.
> 
> 
> Quite so.  Unfortunately, this reviewer sees a lot more poorly
> designed simulations being written up and submitted to periodicals
> than poorly designed laboratory experiments (or, for that matter,
> papers with conclusions based on incorrect maths).
> 
> I think the root problem here is that many universities
> are encouraging students to run simulations without teaching
> students about how to use simulation properly and without
> sufficient supervision that the simulation was used properly.
> This is unfortunate.

And this is exactly what I complained about.

Moreover, it´s an inacceptalbe waste of time and money. How long does it 
take for a gifted student to get _really_ familiar with the NS and the 
basics of simulations? At the risk that I have to ask my provider for 
additional disk space for my mailbox and that I will have learned at 
least two hundered new swearwords by the end of this day: I don´t 
believe that even an excellent student will really _understand_ what he 
is doning in less than one year. Assumed, the student is engaged in the 
NS _full_ _time_. I do know some PhD students engaged in "projects" 
(this are these funny things to make your department "visible", in real 
life, it´s called "advertising") the major part of their time. And even 
with the best effort, they don´t achieve useful simulations. They 
achieve pure botchery. And they can´t even help it because the day has 
only 24 hours.

Yesterday we talked about Tcl here. I did not fully understand the 
arguments in favour of Tcl yet. However, I´m not fully convinced that 
the mixture of Tcl, OTcl, Tclcl and C++ is really that good idea here.

Even this mixture is extremely error prone (I once asked a colleague who 
was thought to be a brillant expert for the simple reason, why a 
Trace/Deque is Trace/Deque and he smattered a lot of bullshit but _NO_ 
_ONE_ could explain me what I already guessed that time, that for the 
split object the initialization code for a Trace object is executed only 
for the reason of the notation "Trace/Deque". This appears plausible. 
But I have looked for this in numerous documents for nearly a year to 
make it sure, not full time of course, otherwise I could have read the 
whole Tcl interpreter). It is extremely difficult to understand.

And it´s a nuisance to debug.

I spent many hours in the debugger to get problems in the C code fixed 
and it´s always nasty when you have to step over a "tcl.evaf()" and have 
to treat it as a black box. Particularly when you run in problems with 
dangling pointers or null pointer exceptions or something similar.

Perhpas, I don´t have the right intuition for the big picture here, but 
for me the NS is a toolbox, a framework. And if I had a faster computer 
and not have to wait hours for the link run, I would appreciate doing 
the whole thing in C++. Perhaps, I´m not an experienced programmer, but 
I´m not convinced of having the need for an embedded language. (And I´m 
old fashioned as well, I strongly prefer C++ over Java, but this is my 
personal attitude. One single reason: Java does not have pointers. And I 
don´t know a language, where a similar depth of knowledge and experience 
with pointers is required as in Java, only to give yourself the illusion 
you would not use pointers. In consequence, I´ve never seen that much 
null pointer exceptions and pointer errors as in Java elsewhere. We 
should not avoid pointers, if we can´t, but learn to use them.)

When I said, it takes a year to understand the ns, at least nine month 
of this is spent for understanding the mixed language stuff. And the 
bindings. And the slit objects. And what is initialized where and when.

Other people make a baby in this time.

So, finally a student often simply wastes a year only to understand the 
ns. And this should be drastically shortened because no one has the time 
and the money for that.

The ns itself is basically a framework and other frameworks can be 
understood in a shorter time too.

The important difference to other frameworks is that in the ns, you are 
often required to work at the framework itself. It is not really 
possible to take the C++ framework as a black box and do some Tcl 
scripting. In fact, I would prefer to do that little Tcl sripting I have 
done in C++ as well.

Who complained recently, his students did forget the binding?

God in Heaven! If we did not do this mixed language stuff, we would not 
have these pitfalls!

I think, this is simply cumbersome and nasty.

> 
> I'm not sure what is available in the line of standard textbooks,
> but if one doesn't yet exist maybe there would be room for someone
> to write up a textbook on when and how to simulate communications
> systems -- or maybe even a text on good experimental design
> more generally.
> 
> Yours,
> 
> Ran
> 



-- 
Detlef Bosau
Galileistrasse 30
70565 Stuttgart
Mail: detlef.bosau at web.de
Web: http://www.detlef-bosau.de
Mobile: +49 172 681 9937