[e2e] Is sanity in NS2?

[Roland Bless]

Hi,

Erik Nordström wrote:
> I totally agree with the statement above. Comparison to reality is the
> only way to do validation.

Yes, that's why we ported a real FreeBSD TCP/IP stack into the OMNeT++
simulator. And guess what: it works very well.
See http://www.omnetpp.org/article.php?story=20050702071848785
for more information.

Background:
Some years ago I started to look for simulators and found ns-2. However,
at this time it was even so complex to understand that I turned to a
(IMHO much) better alternative:
OMNeT++ (http://www.omnetpp.org/).

It is open source and free for academic use. It is written in C++, very
well-structured and easy to learn (comes with a comprehensive
and easily readable documentation). It is also more generic than
NS and offers a better scalability. Furthermore, you don't have to
program in OTcl, which I also found as an obstacle to learn ns-2.
Some people mentioned to rewrite ns-2 to from scratch. I disagree.
People should have a look at OMNeT++ and give it a try...

A disadvantage of OMNeT++ is, however, that it has far less models
available compared to NS-2. The existing TCP implementations seemed
to be bogus at this time, and, getting all bugs out of a full-featured
TCP implementation is really not easy. Therefore, we decided to use an
existing TCP implementation in order to integrate it into the simulator.
(There is also a WSC paper how this were actually done:
http://doc.tm.uka.de/2004/bless-doll-omnettcp-wsc2004.pdf)

> That brings me to another point. Why do people abstract away more than
> necessary in simulators? Why can't ns for example use real packet
> headers and treat packets in the way a real OS does it? If people claim
> the reason is for efficiency I can point out that the current way ns
> deals with packets and headers is not exactly optimal. If ns would
> implement or mimic real API:s (e.g., berkeley sockets) to the extent
> possible, without sacrificing too much performance and scalability, it
> would be much easier to develop protocol code that can run both in
> reality and simulation. That way validation with reality is so much
> easier and trust in the code (that it does the right thing) would
> increase tremendously.

Yes, that's what OppBSD can do...

> For my own implementations, I have gone to great lengths to make sure
> that my code runs both in reality and in ns-2 (without emulation layers
> or similar). This has made it possible to deploy the same code in real
> testbeds and has allowed us to run real experiments and simulations in
> parallel. This way we have discovered many false assumptions made in
> other research that rely solely on simulations. It is also  possible to
> do trace based simulation instead of using the de-facto standard ns-2
> randomly generated scenarios that have little resemblance to reality.
> 
> I think simulators are heavily over-utilized in our reasearch field,
> simply because we are geeks that are too lazy to do real experiments. I
> think we should put our efforts in developing good methodologies for
> conducting repeatable real world experiments instead. Simulation should
> only be used where and when it can be a complement.

Hmm, I think that it is still necessary to do simulations, especially
if you want to analyze scalability issues in scenarios with several
thousand nodes (which I use in OMNeT++).

Validation of the implemented protocols is however important and
the hard part of it.

Regards,
 Roland