[e2e] end2end-interest Digest, Vol 19, Issue 11

[Jim Gettys]

On Thu, 2005-09-15 at 06:16 -0400, David P. Reed wrote:

> First, simulations provide much more efficient experimental 
> environments.   It's very hard to construct repeatable experiments in 
> vivo (so to speak) and in many cases the most important in vivo 
> environments are inaccessible to the researchers who have the time and 
> insight to explore the range of possibilities.
> 

Having once been crazy enough to perform such an experiment (in Vivo
testing of HTTP/1.1 across the continent on the live internet), and also
to have used some simulated behavior for the same experiment (using
nistnet to provide controlled latencies and bandwidths in a controlled
experimental setup), I can attest to the fact that the experiment on the
real network is about 10 times as hard as the controlled experiment.

Getting repeatable results on the real network (that we could actually
believe) took a couple months of tweaking and analysis, and uncovered
behavior that we had not expected (and needed to understand), and would
not have seen in the controlled experimental environment.  It also
uncovered quite a few bugs in various vendor's TCP of the era.

Arguably, the in vivo results were 10 times as valuable and convincing
as the simulated results, and allowed us to believe simulated results
without much question in the other environments that we had not tested
"live".  At the end we ended up with great confidence our numbers were
not taken from thin air, but reflected reality.

				- Jim