[e2e] end2end-interest Digest, Vol 19, Issue 11

[S. Keshav]

Detlef, 
    Since you mentioned my web site, I will save others the effort in wading
through it to post the single relevant sentence: "The goal of simulation is
intuition, not numbers," R.W. Hamming. I was taught this by Sam Morgan at
Bell Labs, who heard it from the horse's mouth.

In terms of simulator validation, I can tell you how I validated REAL when I
first wrote it in 1988 (BTW, this was not my idea: I was only carrying out
Scott Shenker's instructions): I wrote the same simulation using two
packages -- CSIM and NEST (which eventually became REAL). Then, I compared
every packet transmission and reception at the time granularity of one
microsecond. If there was a difference, I found and fixed any bugs. This
allowed me to find several bugs in both simulators.

Sam Morgan did not trust REAL, and he spent a few months comparing REAL
results with queueing theoretic results for M/M/1 and M/D/1 queues.
(Thankfully, he did not find any bugs.) I wonder if any other simulators
have been compared using this straightforward technique.

My two cents:

Simulation results that do not include

* an analysis of parameter stability, i.e. the length of time you need to
run the simulations before the metrics achieve their steady state value, and

*  both means and standard deviations (or error bars)

are just plain bogus.

I was surprised to find that of the 44 'good' papers I taught last year,
only ONE had results standard deviations. All the rest that had simulation
resuts showed a single data value. Imagine what the situation is for 'not so
good' papers!

keshav