[e2e] Is sanity in NS2?

[Roland Bless]

Detlef Bosau wrote:

> Let´s look what you have now: You hava an OMNET++ simulator with a BSD
> stack.
> And hopefully, you get a paper from that, good luck!
> 
> However, you replaced one simulation by another. Did you by chance read
> my reply on George´s post at the end of last week?

It was rather lengthy, but I surely agree on the fact that
you must decide up to which degree of abstraction you want to
model your problem. If you don't care about TCP
dynamics/behavior/performance etc, then you probably don't need
to simulate TCP below your protocol.

The simulator is indeed only a tool to run the model, but as in
reality: a good, high quality tool makes life easier.

> IIRC, I´ve written there, that´s no problem to implement TCP into a
> simulator. And btw: I don´t want to discuss whether Linux is reality or

It doubt that, since it's really a lot of effort to get it right.

> BSD (of course, BSD is considered the reference implementation), I
> personally have great respects for _standards_. So, a TCP implementation
> should always be done
> RFC conformant. 

Yes, but besides the fact that it's even not easy to determine which
RFCs are relevant (cf.
http://www.ietf.org/internet-drafts/draft-ietf-tcpm-tcp-roadmap-04.txt),
currently the IETF doesn't define conformance tests (which is ok for
me). So especially, if you want to analyze TCP performance, you need a
very detailed TCP model and you have to test nearly every particular
externally visible  TCP behavior (like response to different loss
sequences etc.). This would mean to run and evaluate many test
scenarios, which would be different from running interoperability and
functional tests only.

> BTW: IIRC, a great deal of the NS2 implementation of TCP was done by
> Sally herself. So, the person who wrote the standards and the person
> who proposed TCP/Reno and the person who wrote the code were identical.
> That´s great! Why did you write a new implementation?
> And this is the normal situation. When I worked with TCP/Snnop, I
> compared Hari Balakrishnan´s code for BSD and the NS2 more or less line
> by line.
> And believe me: Despite the parts where the NS2 and BSD are that
> different that different coding is _required_, I think the same problem
> arises
> in OMNET++, the code is identical. Great!

I don't get that point.

> However, my point was not to discuss TCP implementations. This is
> implementation work. I´m interested in the behaviour of wireless
> networks.

Yep, but if you want to look at TCP performance aspects in wireless
networks, a quite detailed TCP model is necessary. Thus you need
a heavily tested implementation (I don't doubt that the ns-2 TCP
implementation is quite good and widely used).

> details of network simulators. However, it´s concerned with
> architectural issues in TCP, and therefore the question whether a
> wireless
> link raises architectural issues by it´s nature is stronly on topic. 

I don't know exactly what you're interested in, but I think there has
been a lot of research in this area (output like RFC 3522, 4015 etc.)

> So, the discussion is: How do we simulate a wireless link? And the only
> way to validate a model for that is to compare a model with reality.

That depends on what you want to look at. So if you model not
every detail, but the important ones that you're interested in,
then that might be ok. However, like with mathematical models,
you must know what simplifications are reasonable and will
likely not affect the qualitative outcome etc. the same is true
for reasoning used traffic patterns etc.

> Of course, they are situations where the whole overhead of a simulator
> is not necessary and I can do the implementation work myself.
> However, this raises another question we talked about last week: Who
> will ever believe me? Jon argued, that first the code should
> be made available, then we may publish the papers. From this point of

The paper should describe all necessary details about the model and
assumptions as requested in the paper from Pawlikowski et al. about
credibility of simulations:
On Credibility of Simulation Studies of Telecommunication Networks,
IEEE Communications Magazine, January 2002, pp. 132--139

> view, the ns2 is basically a commonly accepted and commonly used
> simulator which is pretty well known to the vast majority of researchers
> in this area. Even more, you will find that a lot of source code
> was contributed by the authors of the original papers and PhD theses.
> This is one of the particular strength of he ns2.

Yep, agreed, and that was once a main motivation to have ns-2, right?
Its usability is however not always acceptable.

>>Hmm, I think that it is still necessary to do simulations, especially
>>if you want to analyze scalability issues in scenarios with several
>>thousand nodes (which I use in OMNeT++).
>>
>>Validation of the implemented protocols is however important and
>>the hard part of it.
> 
> 
> What do you want to validate here? Do you want to validate that the
> protocols are logically correct? This is one point. Admittedly,

Yes. The protocol should behave like it was specified. But the other
point is: if you're interested in performance/scalability etc. then
you need a quite detailed model.

> I don´t know whether a network simulator is the right tool for this or
> whether you would better use tools for temporal logic or something like
> that.

No, I don't use simulation for prooving protocol correctness.
Simulations are usually good to investigate the qualitative
behavior of a system under variation of certain parameters.

Regards,
 Roland