[e2e] Is sanity in NS2?

[Erik Nordström]

Having been an e2e reader for some time (but infrequent poster) I think 
I can contribute something to this thread.

Rik Wade wrote:

>On Thu, September 15, 2005 02:59, Detlef Bosau wrote:
>
>  
>
>>The best, and basically the only _real_, validation is to compare
>>simulation results with reality.
>>    
>>
I totally agree with the statement above. Comparison to reality is the 
only way to do validation.


>And we all know how accurate and RFC-compliant real-world protocol
>implementations are. I've worked in a few large ISP/telcos and cannot
>  
>
Reality consists of the world around us, and as far as I know it is not 
defined by protocol implementations that run in operating systems. 
Reality is what simulators should simulate, for example radio 
propagation, not some invented artifact.

Having some experience from implementing two ad hoc routing protocols, I 
can say that my primary usage of simulators such as ns is for 
prototyping and testing. It is very convenient to test the logic of your 
protocol implementation in a simulated environment because running 
around, for example, in a corridor trying to set up multi-hop routes 
with laptops is very tedious. I think we should realise that this is one 
of the main applications of tools such as simulators. We should stop 
using ns-2 for generating last minute results for those mandatory (but 
seldom useful) graphs at the end of our research paper.

The other thing you can use simulators for is to achieve scalability. 
Tests that are simply too large and complex to conduct in reality are a 
clear application for simulation. However, if an experiment can be run 
on a smaller scale on real hardware, I see no excuse to use a simulator 
other than simply as
a tool for debugging and development.

That brings me to another point. Why do people abstract away more than 
necessary in simulators? Why can't ns for example use real packet 
headers and treat packets in the way a real OS does it? If people claim 
the reason is for efficiency I can point out that the current way ns 
deals with packets and headers is not exactly optimal. If ns would 
implement or mimic real API:s (e.g., berkeley sockets) to the extent 
possible, without sacrificing too much performance and scalability, it 
would be much easier to develop protocol code that can run both in 
reality and simulation. That way validation with reality is so much 
easier and trust in the code (that it does the right thing) would 
increase tremendously.

For my own implementations, I have gone to great lengths to make sure 
that my code runs both in reality and in ns-2 (without emulation layers 
or similar). This has made it possible to deploy the same code in real 
testbeds and has allowed us to run real experiments and simulations in 
parallel. This way we have discovered many false assumptions made in 
other research that rely solely on simulations. It is also  possible to 
do trace based simulation instead of using the de-facto standard ns-2 
randomly generated scenarios that have little resemblance to reality.

I think simulators are heavily over-utilized in our reasearch field, 
simply because we are geeks that are too lazy to do real experiments. I 
think we should put our efforts in developing good methodologies for 
conducting repeatable real world experiments instead. Simulation should 
only be used where and when it can be a complement.

Erik Nordström