[e2e] Is sanity in NS2?

[Detlef Bosau]

May I first, once again, refer to the quote on Wesley Eddy´s Homepage:


"Mathematicians stand on each other's shoulders while computer 
scientists stand on each other's toes." -- R. W. Hamming


Jon Crowcroft wrote:
> I think the open source/commons implication is part of the problem, but also
> the success disasster that NS became is partly because they made it too easy 
> (analogous to socket programming)
> so anyone who could wield a bit of tcl  or modify an existing bit of C++,
> thought they could build an interesting new simulation, and didnt have to acquire
> any disciple to do so - some people (typically hard put upon PhD students either
> strongly self motivated and sef-disciplined) managed to do some useful things, but
> an awful lot are those hotmail/yahoo email folks you allude to, 
> (I have no idea where they are really from - are they using such
> addresses because they are afraid their university will catch them plagiarising, 
> or are they blocked in china?) and they do more harm than good.

It´s a difficult issue.

First of all: I don´t think that something must be complicated in order 
to be good. I don´t believe in the "Quality by obscuracy" principle.

(I must admit: I never submitted something to the IOCCC.)

I´m a strong believer of clarity.

Unfortunately, the clarity of the NS code is somewhat different in 
different parts of the NS.

Concerning my own NS code: This mostly consists of evil hacks, as I´m a 
bad programer. If someone wants to have a look at it (at his own risk of 
course and I take no responsiblity for any consequences such as brain 
damage, loss of any kind of programming expertise etc.) he can have a 
look at my homepage: http://www.detlef-bosau.de/ns-2.26-cvs.2005.03.15.tgz

It´s not documented in any way. Don´t look for PTE. It´s not that I´m 
not interested in PTE anly longer. It´s because much of the ideas came 
to me when my chair got broken and so, I had to go to the IKEA furniture 
shop (is this correct? I did not find it in my dictionary.) to buy a new 
one. Therefore, it´s called "ikea". However, I could not use this name 
for copyright reasons, so I renamed my approach to "Path Tail Emulation".

Basically, open source programs may work fine. Take Linux as an example.

However, contributing to a large program requires knowledge and self 
disciipline. (I lack both of it....) Particularly for the NS, it 
requires a very deep and very broad knowledge of programming techniques. 
And I´ld better not talk about my first steps into the NS2 world. You 
would roll on the floor laughing.

The problem was: There was no one to teach me.

So, basically I believe into open software and open source programs. 
However, this requires two necessary things:

1.: An excellent maintenance of the program. I think, Linux Torvalds, 
Lars Wirzenius etc. spent at least that much time for maintenance and 
code integration than all contributors together spend in programming. 
(It´s a guess, I did not talk to one of them two.)

2.: An excellent programming education. That´s a matter of course for 
computer scientists. And if there is any school with a different 
attitude, it´s clear what must change there. A computer scientist is 
bascially an engineer, the computer is his engine and programming is his 
  hammer and his screwdriver.

Of course, some programmers are more gifted than others. However, a 
computer scientist is an _excellent_ programmer anyway - or he is no 
computer scientist.


> 
> anecdote - about 6-7 years ago we went through a lot of papers on tcp/aqm that
> had used ns (on the order of 100 papers) and tried to find the ns code to see if
> we could reproduce the results - around 50% of the papers appeared to be either bogus (when you 
> got th code it didnt work with the alleged version of ns) or appeard to have graphs generated
> from a single run of ns (thre first one). quite a few were based o na (short lived) bug in the 
> congestion control code in ns, (EPFL and others confirmed this bug - i cannot remember the exact detail, 
> but it meant that the otcl  and c++ variables weren't bound, so if you varied one, the other one didnt -
> this meant that the results for cwnd were basically random - papers were publised on this too).
> 

The _serious_ issue behind this anecdote is: Who has ever validated NS2 
code? How much papers are based upon "irreproducible simulations"?

In consequence, I think: Simulation proves anything and nothing. It´s 
basically what Keshav mentions on his homepage. It´s the _rationale_ 
that must be convincing. Not one simulation.

Simulation results may result from a "lucky choice" in your scenario or 
from sum "lucky bugs".

You might kill me for that example. But when I think of the TCP-AP 
approach, Mobihoc 2005, ElRakabawy, Klemm, Lindemann, I simply don´t buy 
it. I contacted the authors and asked for the rationale of this 
apporach. I don´t got one. The authors refered to the simulations.

First: I don´t have the source.
Second: I don´t have the time to analyze each simulation artefact.

This paper "proves" its approach with "a pletora" (i.e. two) scenarios 
and no rationale. I don´t understand, why this paper was accepted. It´s 
an embarrassment for the whole community.

I conducted a similar approach years ago - and I had to learn it was 
wrong. It was a bitter lesson. You can imagine my aggravation.



> before ns, there was XSim, and real and the other descendants of the MIT simulator, which had a similar series of
> problems, although were sufficiently hard to use that most people ended up doing pretty much clean 100%
> rewrites of the relevant part of the code for their thesis work, and ended up
> i) understanding it
> ii) validating it
> iii) getting lots of meaningful results out
> iv) abandoning it completely upon graduating...

The pity is iv.

That´s why I wrote the qote above.

It´s hardly possible to reinvent the wheel each time you do a research. 
And it´s a validation matter.

Why did I ask for AWND and flow control?

Because I want to compare my own PTE approach with a "dumb spoofing 
approach" (look at Mark Allman´s work on that one).

It´s basically no problem to introduce a "dynamic AWND" into the NS2.
It´s the problem to have the TCP sender react properly particularly on 
zero windows. In addition, a split gateway or a PEP using PTE has to 
obey some rules for the proper definition of the AWND.

It´s not that I don´t want to do this myself (although I would be glad 
to use existing code here) or that I was not able to do this myself. Of 
couse I´m able to.

It´s a matter of validation.

And I´m afraid that a submitted paper would be rejected, because the 
reveiwer simply does not believe me or the scenario is not "complex" 
enough or whatever excuse is used.

To be honest: I´m still not sure, wheter I will use a simulation in a paper.


> 
> 
> Your comments about opnet also apply to matlab and other propietary and quite good (or very good, respectively)
> tools, that are supported so long as the relevant supervisrs ask for funds - they are relatively expensive for 


The problem with proprietary code is that I cannot validate it. I cannot 
  read the code. I have to "believe" it.

> UK university budgets so are typically default-off - which is ludicrous really given the time it can save a student
> and supervisor and quality improvement in results....
> 
> we attempted to do an NS re-write in java (jns and jvs) which worked pretty well and several folks picked up on it,
> but exactly the same thing started to happen to us (at UCL) so I abandoned the program of work (though others
> picked up on it and it is on sourceforge i believe (not due to us) but i dont know how active it is -
> 
> one specific thing that attempted to do was to make it "proper programming" to use the simulator, 
> so the tecchnical bar was set a bit higher than just throwing some modified tcl at something and hoping...

I basically don´t believe that making thinks more complicated would be 
more helpful.

In my opinion, the main issue is discipline and proper maintenance. 
Particularly, it´s a bad idea to ad arbitrary "contributors" on the NS2 
homepage without any consideration if
1. the contributed code is _relevant_,
2. the contributed code is _correct_,
3. the contributed code conforms to certain standards which consider the 
quality of code as well as some style rules.

AFAIK, in commercial companies you have programmes who _intergrate_ 
contributed code. Who rework the programming style, who care for proper 
documentation etc.


> so trying to counter the point i made above by setting an implict "qualifying exam" to drive jns -


Nice idea!

> but i think its evidence that at least in the case of educational software, open source may not be a good idea...
> supporting your point again.
> 

At least for me, I´m an unemployed person, there is no alternative to 
open source software.


> agree with your points on support mail lists etc too
> 
> in fact i think this is the nearly first time i agree  100% with what you wrote!

We all are happy for both of you :-)

Refering to Hamming once again: Feel perfectly free to continue to do so :-)

(And yes, someone told me that a book on the NS2 would be helpful.... 
However, at the moment I don´t dare to start a project like this. First, 
I don´t know wheter I´m able to do so. Second, there are other things 
more important, e.g. to find a job in Germany at the age of 42. Perhaps, 
I should apply as a test person for coffins. However, this is not 
adequate for a person interested in packet switching, as undertakers 
usually prefer to do line switching.)

(And please, don´t sent me comments on my programming style. I know, if 
I ever wrote a book on the NS2, I would look for a co-author who is a 
gifted programmer....)


Detlef


-- 
Detlef Bosau
Galileistrasse 30
70565 Stuttgart
Mail: detlef.bosau at web.de
Web: http://www.detlef-bosau.de
Mobile: +49 172 681 9937