[e2e] Is sanity in NS2?

[Detlef Bosau]

George Riley wrote:
> 
> I've restrained from replying until now, but must jump in.
> 
> Yogesh Prem Swami wrote:
> > I have a somewhat general question about simulations. My question is
> > that is there any scientific reason why simulations should be able to
> > predict the behavior of a real packet transmission phenomena? Unless
> > you
> > make the assumption that packet transmission/interaction is a
> > non-chaotic phenomenon (chaotic, as used in physics), there is no
> > reason
> > to believe why a simulation would be able to model real world events.
> 
> What chaotic behavior are you talking about?  The behavior of a TCP
> endpoint is completely deterministic.  Given a set of data demands on


It´s great :-)

I´ve heard countless times, sometimes enriched with an arrogant "as we
all know": "The Internet is self similar."
Not that I´ve heard a proper definition for self similarity each time.
And typically, self similarity, long range dependency and chaotic
behaviour are mentioned together.

I don´t have any knowledge about this, but I´m always curious when I
read this.


> a TCP connection, and a given network topology with no competing
> flows,  the endpoint will do exactly the
> same thing every single time.  The behavior of a router with a fixed
> size
> DropTail queue and a fixed set of links will behave exactly the same way
> every time, given an arrival pattern of packets.  Both of the above
> statements
> are ignoring small variations in interrupt response times and process
> scheduling
> at the systems in question, but these variations are small and can be
> ignored
> without affecting measured results.

George, did you by some chance read my remark that the NS2 is for
networks what Google is to reality?
Did you _count_ the number of assumptions you made?

I absolutely agree with you: If we made the network to behave exactly
like the NS2, the NS2 would yield a perfect network simulation.
However, we make quite a few abstractions here. And of course, many of
them are justified. However, many are questionable. 

Perhaps, it´s time to refer to my favourite paper here once more :-)

@article{martin,
journal = " IEEE/ACM TRANSACTIONS ON NETWORKING",
volume ="11",
number = "3",
month = "June",
year = "2003",
title = "Delay--Based Congestion Avoidance for TCP",
author = "Jim Martin and Arne Nilsson and  Injong Rhee",
}      


One important point made by the authors is, that they simply _failed_ to
reproduce the behaviour on L2 as it was measured in real networks.
I did not follow the details, but I think they replaced the link model
by something which reproduced the pracitcally measured observation.
This perfectly corresponds to Kathies statement, that it is difficult to
extrapolate from the typical "mini scenarios" used in the NS
(Dumbbell, Dumbbell++, Dumbbell light...) and perhaps some funny WWW
simulations with even 1000 nodes in it to the global Internet.

Moreover, I´m interested in mobile wide area wireless networks like UMTS
or GPRS. And at least the _physical_ constraints in a radio channel
can hardly be reproduced. And I´m looking eagerly for appropriate
models.

However, basically the whole thing very much depends on what you want to
simulate. So to say: An answer is hardly better than its question.
Some of my beloved reviewers told me to make the scenario more complex,
to add cross traffics etc. etc. etc. 
Why? Of course, I have to "simulate all scenrios, Mr. Bosau, it´s a
matter of industry". 

The number of scenarios in the Internet is infinite. So, the old saying
becomes true: More is less. And the art of simulation is the art of
abstraction and simplification. I´m convinced that a proper posed
question can be answered well with a proper chosen scenario.
However, even in my PTE work, I don´t want to simulate "the Internet". I
look for answers for very special questions.

However, I do abstraction and simplification here. I found David Reed´s
remark helpful here. Newton´s Principia as the "apple simulation". 
Sounds funny. But it´s the basic modeling approach taken in physics! And
we know, that the Principia is an abstraction. As such, 
it does not explain "the world", but it sheds light on a number of basic
questions.

> 
> The job of the simulator is to recreate these deterministic and well
> known
> behaviors, without having to construct a real network.  A high-quality

Well known?

Again, I´m dealing with wireless networks. From what I have read during
the last five years, the only well known fact we know about mobile
wireless
networks is that we hardly know anything! And the situation is not
better in wirebound networks. Take for example Keshav´s paper on the
Ethernet
capture effect. Do we find this in the NS? I don´t know, I only ask.

So, what we recreate and simulate _is_ already an abstraction.

And as such, it is justified. 

Abstraction and simplification is not a bad thing per se. Although the
Principia is an abstraction and simplification it was sufficient to
land a man on the moon and return him safely to the earth. 

The art is to do the _right_ abstraction and the _right_ simplification
here.

Sometimes, I see the other way round. Particularly some Emulation people
tend to reproduce the network as closely as possible. Frankly: I don´t
expect that we ever will achieve this goal. And I don´t see a sense in
it. The best emulation of reality is _reality_. But the way to
_understand_ reality is the art of 
abstraction and simplification. And the continuous dialog between
simplification and reality: Can we make proper forecasts? Do our
extrapolations hold in reality? That´s the way we judge our
abstractions.



> simulator, such as ns2 or GTNetS does exactly this.  A competent
> programmer
> can read the RFC's describing what the TCP endpoint is supposed to do,
> and implement a model that does exactly that.  BTW, if you haven't seen
> the Georgia Tech Network SImulator (GTNetS), it might be worth a quick
> look.  Google for GTNetS and check it out from our CVS repository.

Question (Before I download it and compile it here): How do you simulate
a mobile wireless network? How do you abstract, e.g., GPRS?
What I´ve seen so far in NS2 is no abstraction but the attempt to
simulate each clamp and every cable. 
Can I rely upon your GPRS simulation in that way, that your simulator
properly _predicts_ a behaviour for TCP which can be reproduced in
_reality_?

> 
> The above statements are bit tongue-in-cheek of course, because we all
> agree that networks are in fact chaotic.  But where does this chaotic

"As we all konw."

Do we know it? Or do we agree upon it? Or can you give a _rationale_,
why we expect this? One of my teachers at the Technical University
Braunschweig / Germany told me that for asserted "randomness" it´s
always important to know, where the randomness stems from.

> behavior come from?  It comes from unknown and unpredictable behavior
> of the end-system applications and users.  Again, any good simulator can

Really? In random distributions we often observe asymptotic behaviour.
(Law of Large Numbers, CLT, Chebyshev´s inequality etc. etc. etc.)
So, even if we cannot predict the results from some individual
experiment we can do statistics. Why do networks behave different here?
Or: Do they? I don´t know. But e.g. Sireen Malik argued using the CLT
some few weeks ago.


> model this as well, as both ns2 and GTNetS can do.  The randomness


It´s not the simulator, which does this. It´s our _model_. Our
_abstraction_.

When I played around with some latencies here during the past few weeks,
I wrote a little C program for that. And I can imagine
writing some small program for certain purposes when I want to avoid the
overhead of NS2. It´s not the simulator. It´s the _model_.
Concerning the TCP agents etc., it´s quite simple to implement them
properly. Take the RFCs and do the work. And it´s easy to compare
your simulated TCP with "reality", at least as it is intended: Compare
the code with the RFCs.

But how do we simulate a wireless channel? How do we simulate a user´s
behaviour? The _simulator_ will do, if the researcher´s _model_
will do. The hammer will do - if the _carpenter_ does his work properly.

It´s somewhat similar to the old debate whether we should use C++ or C#
or Java or ADA.... Good programming depends primarily on the programmer.
Not on the language. A language may have its advantages and I strongly
prefer C++ over FOTRAN IV. However, it´s a weak argument as I´m a 
bad programmer.

I know people who do "emulation" and think it´s worthwile to emulate 64
nodes or 640 nodes or 64000 nodes. And when I attend their 
talks, I see slides, hear endless debates about engineering. But I don´t
hear anything about abstraction, reduction, simplification.
The strength in Newton´s principia is the simplification and
abstraction. _That´s_ what makes science different from engineering.

David Reed is absolutely correct here. What is the difference between
modelling a network with a discrete event simulator
and numerically solve a systems n-dimensional differential equation
using a Runge-Kutta algorithm? It´s exactly the same.
And the implementation is enginieering.

But it´s _science_ to get the correct equations.

And simplifactin _is_ an art. I think, it was Michelangelo who said it´s
not the problem to build a statue. The statue is already there.
It´s the problem to take away the superfluous material. To remove the
_right_ amount of marble from the _right_ place.
So for GPRS and UMTS, we should leve out these dozens of classes for
each clamp and each cable and all this unnecessary trash.
The art is to leave the unnecessary things and to keep the correct
abstraction and simplification. And then put it into your
prefered simulator and it will certainly do. Like a Runge-Kutta
algorithm which runs on an IBM machine as well as on an HP machine
or a SUN. If you do this correctly, you can leave out nearly the whole
world and keep an apple. And you will land a man on the moon
and will return him safely to the earth.


> is due to randomness in the data demand at endpoints, ignoring
> intentional randomness is certain AQM methods such as RED.
> But again, the job of the simulator is NOT to intentionally produce
> chaotic behavior in a network, but rather to predict what that network
> would do, given a set of (possibly random) inputs.  I claim that a good
> simulation tool can and does do exactly that.

I don´t agree here.

I absolutely don´t agree.

The job of the simulator is to execute my model, as the job of the
computer is to execute my program.

The _problem_ is to build the correct model

The problem is not to build a simulator or to implement the algorithms.
Anyone could do this. And as we discussed during the past days, there
are numerous simulators around. Yesterday we learned from this
wounderful slideset that there has been one in FORTRAN in the 60s.
Your input is the model. And if your model of, e.g., chaotic behaviour
is correct, your results will be. And if your model is wrong, the
results will be.
So basically, it´s a model based prediction. Like a predictor corrector
approach used in numerics ;-) With the only difference, that our models
are the predictor - and reality is the corrector. And it´s one of the
aims in science to make corrections as small as possible in the long
run.
In physics, in weather forecast, in network *mulation.

Of course, your simulator is part of the model. Or in other terms: It
makes a choice for a certain class of models. E.g. the ns2 is a discrete
event simulator. So you model your network behaviour as a sequence of
events happening on discrete points in time. And there is no concurrency
in it. And we always can
put the events in a certain order and get the correct results. It´s
again similar to solving differential equations. Perhaps one of our
older
colleagues can tell us how differnential equations were solved on
analogous machinery. This was _continuous_. Whereas todays numerics is
_discrete_.
It´s perfectly the same.

Even in the ns2 it´s obvious: In Tcl, you plumb together your "model"
(which of course only represents the topology and a number of scheduled
events)
and afterwards, the simulator runs, executes, the so defined model. This
is exactly the same approach as in simulated annealing and all the
thermodynmaic approaches. I don´t think it changed very much during the
last centuries. We only are lucky today to have the machinery to do
the calculations. But the idea is the same as with the iron ring which
is heated until it is white heat on the one half and it´s still cold
on the other. Afterwards it´s covered with earth and now we observe what
happens. AFAIK, this Gedankenexperiment is due to Jean-Baptiste Fourier
and was the motivation for his mathematical work on the Fourier
Transform. Now, we have the discrete Fourier Transform, which is to my
knowledge
due to Kutta (or Runge? I always mix them up) in Königsberg in the
1920s. The principle is all the same.

Today we heat up the ring in TCL, define the temperature, the radius,
the temperature gradient of the material etc. etc. etc.
And then we start the simulator, i.e. we cover the ring with earth and
observe what happens.

Why is this important? What is the lesson learned here?

The lesson is: Your results depend on the _model_. Not on your
simulator. As for the ring, the simulator is only the blacksmith.
In some sense, you will exactly get out what you put in. And in that
sense, a simulator is the perfect garbage in garbage out system.
The same as any computing machinery. (As long as your simulator is
correct, of course.)

Once again for the ns2: Did you ever have a glance at the contents of
the different directories there? Did you ever observe
how many agents and algorithms we have - and how few delay models? And
for the nodes, they typically do not model any delay at all.
So, we have a pletora of algorithms and protocols there. But what about
our _network_ _MODELS_? And what about our traffic models?
And what about our user models? And have the contributed mobility models
ever been checked against reality?
But as I said: That´s not an ns2 problem. It´s the problem of the
models. And perhaps, we simply set the wrong focus here.
When I deal with 3G networks, my main interest is a proper model of the
transport system between Base Station and Mobile.
Not the TCP algorithms. My Goodness, one deals with packets, the other
one with bytes etc. etc. etc. This is possible, I buy that.
But what about the _link_???? Simply compare the length of delay.cc and
tcp-full.cc and you see why I question the focus here. (It´s about a
factor 12. Guess, what´s more complex.)

> 
> Simulation is not supposed to be a replacement for actual in-vitro
> experimentation, but must be used in cases where such experiments
> are not possible.  Consider Sally's recent work on TCP Quick-Start.

Absolutely. 

> The Quick-Start implementation requires support from routers, to
> react to and modify a new options field in the TCP header.  Is Sally
> supposed to somehow get access to Cisco IOS source code and modify
> it to support the new header? Of course not.  Is she supposed to get

Why "of course not"?

Excuse me, but are we doing science or looking for excuses? 
If the algorithm is to be tested on real routers, Cisco could implement
it on an experimental IOS and run it in a testbed. Period.
I´ve worked with poorly tested algorithms in reality and I tell you, I
wouldn´t buy it if it´s not properly tested.

The ns2 model of routers is a rough estimation. We do not (at least to
my knowledge from version 2.26) processing times, no memory access times
etc. etc. etc. 

I worked as a support guy for years. Amongst others in a company which
built computing machinery for midrange data base systems. There, it was 
quite simple: When the customer requested a system which served 5000
clients and the customer wanted a real life test, the customer _got_ his
real life test.

I was responsible for a large customer´s network some years and when I
would have brought such an excuse, and we had experienced
the _least_ difficulty, the customer would have killed me!

It´s of course not Sally´s problem. She can´t help it. But it´s a
general problem and we must accept that networks are used 
in vital systems these days and this is reflected in the user´s claims.

> access to Linux source code (easily done) and hack it to support
> the new option (not so easily done)?  Again of course not.  She uses

Are we talking about Linux? Or are we talking about IOS?

Herer in Stuttgart, some guys assert, Linux is more "real" than the NS2.
This is pure nonsense! Linux is Linux, NS2 is NS2 and reality is
reality. And these three are different by nature.
So we have to decide what we are talking about. You can play around with
Linux if you want so use it at the next Linux installation party.
But if you plan to use an algorithm with IOS on Cisco routers, you have
to validate it in a testbed with IOS and Cisco routers.
I think, there should be no debate on it. This is simply a proper
engineering attitude.

And for Cisco, it´s the reason why at least one company I know refused
to buy these systems, because they had great difficulties
with ISDN which was not tested properly. And when software
"shortcomings" cause a loss of, just to say a number, 1 Million USD a
year for
a company, I honestly tell you: The company was absolutely _not_ amused!


> a simulation tool that she is quite comfortable with, and constructs
> both router behavior and end-system behavior as she imagines it will
> be implemented in the future, and demonstrates the differences in
> performance using Quick-Start.  In this case, clearly a field experiment
> is not possible.

It´s again a matter of modeling. And the model must allow to extrapolate
the result for a large number of routers.
If not: I assure you, the customer _INSISTS_ in a field experiment and I
do understand this.

I think, we must accept the limitations of simulations. And basically:
These are limitations in our models. Once again: The apple was
sufficient
to land a man.....

So, of course, we have to check our models against reality. This might
be quite abstract models. At the KiVS there was a talk given by Matthias
Scheidegger from the University of Bern, Switzerland, Department of
Torsten Braun, who discussed exactly that. Building a _model_ for a
certain portin of the Internet
and put it into the NS2. It´s trivial to put this into the NS2. The
_problem_ is to build the models.

> 
> The criticism that simulation tools, such as ns2, are responsible for a
> plethora of poor quality networking research is just bogus.  The

I agree here :-) And it´s clear, why I do so :-)

> simulator
> is a tool who's job it is to predict what a given network will do under
> a
> given set of conditions.  The researcher is responsible for deciding
> what
> is a reasonable set of conditions to be used to study a given problem.


Not only. The researcher is responsible for the correct _model_.

And in my estimation we put to much emphasis into algorithms and
implementations and far too less emphasis into correct models.
And this is not only making a scenario great and complex but to keep the
important thins and leave out the unimportant ones.

Once again: The art of simulation is the art of abstraction, reduction
and simplification.

> ns2 and GTNetS and the other simulation tools will uphold their end of
> the bargain, it is the researcher's responsibility to uphold their end.

This is correct. But I disagree with your focus. The simulator does not
simulate a network. Neither does an emulator emulate a network.
All this machinery runs models and such is Garbage In Garbage Out by
nature. 
The real _art_ is to build the correct models. To find the correct
abstractions, to find the correct simplifications and reductions.
Anything else is routine engineering and well understood for decades.

Detlef.

> 
> George
> 
> ------------------------------------------------------------------------
> --------------------
> George F. Riley, Assistant Professor
> The School of Electrical and Computer Engineering at Georgia Tech
> Atlanta, GA  30332-0250
> (404)-894-4767
> E-Mail: riley at ece.gatech.edu
> 
> ECE3090 Web Page: http://users.ece.gatech.edu/~riley/ece3090/
> ECE6110 Web Page: http://users.ece.gatech.edu/~riley/ece6110/

-- 
Detlef Bosau
Galileistrasse 30
70565 Stuttgart
Mail: detlef.bosau at web.de
Web: http://www.detlef-bosau.de
Mobile: +49 172 681 9937