[e2e] Simulator for wireless network

[John Heidemann]

On Sun, 15 Apr 2007 21:15:14 +0200, Detlef Bosau wrote: 
>Adam Wolisz wrote:
>>
>>> ALl,
>>> Giuseppe has addressed a very important point.
>>>
>>> In fact the following question is a basic one: what is to be 
>>> investigated - an ideal behavior of the envisioned,
>>> precisely defined solution - or THE behavior of really deployed 
>>> products?
>
>Exactly. And that´s why it´s so important to have the question well put.
>
>And when we investigate the behaviour of a deployed product, we 
>practically investigate the behaviour of _exactly_ _that_ deployed 
>product - which may well be subject to change without notice (as it is 
>written in nearly each manual of any appliance you can buy). Standard 
>violation included, as you wrote later in your post.

I believe this is a key observation, and one that's often node made
explicit in a rush to "more realism" in simulations.  One can only
answer "what's the best simulator" (or more generally, what's the best
evaluation tool) in the context of a specific research question.

If you're trying to do quantitative predictions about a particular
product you'll deploy in a specific place next week, then the best
possible fidelity is essential.

But relax any of those constraints and, in many cases, inaccuracy in
one aspect of the simulation overwhelm accuracy in the others.

For example, running real code in your simulator can be great, but if
you turn around and use an artificial traffic or mobility model,
the results can easilyi be dominated completely by that choice and not
whether you're using mbufs or skbufs, or which bugs you emulate.

This observation motivated the fairly abstract TCP models in ns-2.
Because they're simplified, it is easy to explore variants and
understand the "envelope" of valid TCP implementations, both current,
old, and not-yet-in-wide-deployment.

IMHO, the biggest benefit of simulators that share code with real
implementations is not realism in simulation, but the ability to
iterate between simulation and actual experimentation.

Although a bit dated, some of these issues were discussed in Expanding
Confidence in Network Simulation. IEEE Network Magazine, 15 (5),
pp. 58-63, Sept./Oct., 2001, with a copy at
<http://www.isi.edu/~johnh/PAPERS/Heidemann01f.html>.

   -John Heidemann


(Disclaimer: I was heavily involved in ns development in the past.)