[e2e] Is sanity in NS2?

[Jon Crowcroft]

Lloyd,

this is so on the money in pretty much every comment that I really have to support it.

The story of NS is as 
the story of IP, but sadly, not as
the story of O.

I think the open source/commons implication is part of the problem, but also
the success disasster that NS became is partly because they made it too easy 
(analogous to socket programming)
so anyone who could wield a bit of tcl  or modify an existing bit of C++,
thought they could build an interesting new simulation, and didnt have to acquire
any disciple to do so - some people (typically hard put upon PhD students either
strongly self motivated and sef-disciplined) managed to do some useful things, but
an awful lot are those hotmail/yahoo email folks you allude to, 
(I have no idea where they are really from - are they using such
addresses because they are afraid their university will catch them plagiarising, 
or are they blocked in china?) and they do more harm than good.

anecdote - about 6-7 years ago we went through a lot of papers on tcp/aqm that
had used ns (on the order of 100 papers) and tried to find the ns code to see if
we could reproduce the results - around 50% of the papers appeared to be either bogus (when you 
got th code it didnt work with the alleged version of ns) or appeard to have graphs generated
from a single run of ns (thre first one). quite a few were based o na (short lived) bug in the 
congestion control code in ns, (EPFL and others confirmed this bug - i cannot remember the exact detail, 
but it meant that the otcl  and c++ variables weren't bound, so if you varied one, the other one didnt -
this meant that the results for cwnd were basically random - papers were publised on this too).

before ns, there was XSim, and real and the other descendants of the MIT simulator, which had a similar series of
problems, although were sufficiently hard to use that most people ended up doing pretty much clean 100%
rewrites of the relevant part of the code for their thesis work, and ended up
i) understanding it
ii) validating it
iii) getting lots of meaningful results out
iv) abandoning it completely upon graduating...


Your comments about opnet also apply to matlab and other propietary and quite good (or very good, respectively)
tools, that are supported so long as the relevant supervisrs ask for funds - they are relatively expensive for 
UK university budgets so are typically default-off - which is ludicrous really given the time it can save a student
and supervisor and quality improvement in results....

we attempted to do an NS re-write in java (jns and jvs) which worked pretty well and several folks picked up on it,
but exactly the same thing started to happen to us (at UCL) so I abandoned the program of work (though others
picked up on it and it is on sourceforge i believe (not due to us) but i dont know how active it is -

one specific thing that attempted to do was to make it "proper programming" to use the simulator, 
so the tecchnical bar was set a bit higher than just throwing some modified tcl at something and hoping...
so trying to counter the point i made above by setting an implict "qualifying exam" to drive jns -
but i think its evidence that at least in the case of educational software, open source may not be a good idea...
supporting your point again.

agree with your points on support mail lists etc too

in fact i think this is the nearly first time i agree  100% with what you wrote!

cheers
jon
p.s. another anecdote - the person who re-wrote pim at cisco did work on multicast in ns for his phd.
yes, he found the same problems. so did others. i wish i'd told them to write their own simulator.
mea culpa.



In missive <Pine.GSO.4.50.0509131206000.4689-100000 at argos.ee.surrey.ac.uk>, Lloyd Wood typed:

 >>Detlef,
 >>
 >>There is an ns-developers list, with a much higher signal-to-noise
 >>ratio, and moderation of posts from non-subscribers by Tom Henderson.
 >>But that list is focused on fixing bugs in behaviour in ns (with a
 >>current emphasis from Lacage on rewriting the 802.11 code while making
 >>all other ns programmers look silly), rather than explaining observed
 >>behaviour of ns without reference to code. The pending move to
 >>sourceforge should make more lists and tools available.
 >>
 >>The contact link at the bottom of various ns webpages like e.g.:
 >>http://www.isi.edu/nsnam/ns/
 >>is _still_ the ns-users list address, which is irresponsible, since it
 >>encourages questions without participation or knowledge of context of
 >>the list -- and the after-the-fact autofaq that gets mailed out
 >>doesn't seem to help. This can be considered an example of either open
 >>source trying to externalise its support costs, or the tragedy of the
 >>commons in action.
 >>
 >>These days, everybody is using ns, presumably because it's free, which
 >>arguably is of benefit towards students and their universities. (I
 >>wouldn't want to be a student trying to use Opnet and have the
 >>license expire because academics who don't use Opnet are bickering
 >>about which budget payment for the next year's license should come out
 >>of. Or be considering going part-time as a PhD student and realising
 >>you'd then being exposed to license and server access worries. Or
 >>discovering there's been an Opnet upgrade and the user interface has
 >>changed entirely. Again. In some very important respects, ns gives PhD
 >>students slightly more control over their destinies.)
 >>
 >>I spent a couple of years in the late 90s answering mail on the
 >>ns-users list; I was the first in my university to use ns, so got to
 >>grips with it very very slowly, while others who followed me got on
 >>far faster, since there was some local help for installation etc., and
 >>once over those hurdles they actually knew how to program. (as a
 >>strategy this worked -- I received enough help from others that I
 >>could figure out how to create some trivial solvable problems using
 >>ns, write them up, and graduate. But it's risky and not for everyone.)
 >>
 >>In answering questions on ns (while also asking unanswered questions
 >>more of the form 'these documented commands don't work and ns crashes!
 >>Why?' -- ns multicast is a complete trainwreck, PIM-SM never worked
 >>for me), I have now left myself open to years and years of questions
 >>from yahoo/hotmail-using students who have trouble stringing coherent
 >>sentences together. I imagine it's much the same for anyone else who
 >>has contributed to ns, and shudder to think how much mail e.g. Floyd
 >>or Heidemann must get on a daily basis. This discourages experienced
 >>participation in the ns-users mailing list; after graduation I
 >>realised I had to unsubscribe for my own sanity.
 >>
 >>Yes, a moderated ns-users list would be a good thing, but it would be
 >>a full-time job, and there's no ns-related charitable foundation.
 >>(Although sourceforge can accept paypal donations, so there could be a
 >>way of channelling money... hmmm. What's the business case? What's
 >>the selling point for donations?)
 >>
 >>But complaining about ns misses the point. You may as well complain
 >>about why universities don't teach you practical programming 'here's a
 >>ten-year-old codebase and here's how to figure out how it works'
 >>skills. Universities just expect you pick everything up by osmosis;
 >>hey, get students to spend enough time in a room with computers, and
 >>maybe they'll learn to control them. The principle works with
 >>libraries, books and reading, right?
 >>
 >>Universities are effectively trying to outsource their support
 >>costs, too, in an increasingly short-term world.
 >>
 >>L.
 >>
 >>recommends http://polaris.gseis.ucla.edu/pagre/network.html
 >>
 >>
 >>On Tue, 13 Sep 2005, Detlef Bosau wrote:
 >>
 >>> Please don=B4t wonder when I ask this here.
 >>>
 >>> I placed this question (dynamic AWND) yesterday into the ns-users list,
 >>> after having subscribed to this list again after a break of about two
 >>> years.
 >>>
 >>> I think, I=B4m going to unsubscribe again. The ns-users list is simply
 >>> useless. One of the most important facts in using mailing lists is that
 >>> after the third question like "Urgent!!!!!!!! I lost my shoes!!!!!!",
 >>> "Urgent!!!!! Need help!!!!! I=B4m going to take an exam!!!! Help me!!!!=
 >>!"
 >>> Or "I cannot install the ns2, there are much of this funny files ending
 >>> with .cc, what=B4s hat?", "Where is the ns2 documentation? There are mu=
 >>ch
 >>> of that strange .h files, but they are apparently no _help_ files" etc.
 >>> etc. etc., any user who has in fact an idea of what the ns2 is or what
 >>> programming is all about will simply unsubscribe. In addition, it is
 >>> simply not possible to follow this lare amount of mails there.
 >>>
 >>> Perhaps, one could think about a ns-users list with a better
 >>> administration. At least questions like: "which linux must i install
 >>> on my nt?" or "will nt work in mac when i install my ns in linux?"
 >>> or similar should be made to disappear. It=B4s not only annoying but
 >>> simply impossible to pick out that one useful post of the hundred
 >>> posts each day that way.
 >>>
 >>> >From my own observation, I=B4ve seen that there are universities the
 >>> students of which boast with large ns2 knowlege - however in fact, ther=
 >>e
 >>> is none.
 >>> And there is a clear reason for this: There is no introduction into the
 >>> ns2, no lectures, no ns2 users group etc. etc. at these universities.
 >>>
 >>> In nearly every village in Germany you have a Linux Users Group.
 >>>
 >>> I don=B4t know how this is in Stuttgart but in larger and more importan=
 >>t
 >>> villages like Markl/Inn or Oberammergau you surely find one ;-)
 >>>
 >>> So, if there is some guy having problems with Linux, he can ask there
 >>> and will perhaps get assistance. In addition, one can arrange some kind
 >>> of kknowledge transfer and education there.
 >>>
 >>> Please excuse me, when I=B4m upset on this one. The ns2 is a useful too=
 >>l.
 >>> It is very common and it=B4s, in general, a real good work.
 >>> (It can be seen, however, that many programmers contributed to this wor=
 >>k
 >>> and that not all of them are equally skilled. But that=B4s
 >>> life.) It would be a pity, when a useful and helpful program with many
 >>> men years of work in it and much excellent knowledge therein
 >>> will perhaps become less attractive in the long run, because there is n=
 >>o
 >>> useful venue to discuss ns2 matters.
 >>>
 >>> >From what I=B4ve seen in this one single day, I=B4m about to suggest t=
 >>o
 >>> close the ns-users list and to start a new discussion venue from
 >>> scratch.
 >>> I don=B4t believe that ns-users can be salvaged.
 >>>
 >>> Detlef
 >>>
 >>> --
 >>> Detlef Bosau
 >>> Galileistrasse 30
 >>> 70565 Stuttgart
 >>> Mail: detlef.bosau at web.de
 >>> Web: http://www.detlef-bosau.de
 >>> Mobile: +49 172 681 9937
 >>
 >><http://www.ee.surrey.ac.uk/Personal/L.Wood/><L.Wood at eim.surrey.ac.uk>

 cheers

   jon