[e2e] Is sanity in NS2?

[Sally Floyd]

Ran -

>Unfortunately, submitted papers that both use unvalidated models
>and use them in an unvalidated scenario are all too common.  

The newly-created Transport Modeling Research Group (TMRG) is working
towards creating Best Current Practice sets of simulation and testbed
scenarios for the evaluation of congestion control mechanisms.
These are not intended to be complete benchmarks, but just "you
should have tested with these scenarios also".  Towards that end,
two documents have been written so far, "Metrics for the Evaluation
of Congestion Control Mechanisms" and "Tools for the Evaluation of
Simulation and Testbed Scenarios".

http://www.icir.org/tmrg/

Any feedback and contributions would be welcome, on the tmrg mailing
list.  Abstracts are below, for those who are interested.

Many thanks.

- Sally

Metrics for the Evaluation of Congestion Control Mechanisms:

    This document discusses the metrics to be considered in an
    evaluation of new or modified congestion control mechanisms for the
    Internet.  This document is intended to be the first in a series of
    documents aimed at improving the models that we use in the
    evaluation of transport protocols.


Tools for the Evaluation of Simulation and Testbed Scenarios: 

    This document describes tools for the evaluation of simulation and
    testbed scenarios used in research on Internet congestion control
    mechanisms.  We believe that research in congestion control
    mechanisms has been seriously hampered by the lack of good models
    underpinning analysis, simulation, and testbed experiments, and that
    tools for the evaluation of simulation and testbed scenarios can
    help in the construction of better scenarios, based on better
    underlying models.  One use of the tools described in this document
    is in comparing key characteristics of test scenarios with known
    characteristics from the diverse and ever-changing real world.
    Tools characterizing the aggregate traffic on a link include the
    distribution of per-packet round-trip times, the distribution of
    packet sequence numbers, and the like.  Tools characterizing end-to-
    end paths include drop rates as a function of packet size and of
    burst size, the synchronization ratio between two end-to-end TCP
    flows, and the like.  For each characteristic, we describe what
    aspects of the scenario determine this characteristic, how the
    characteristic can affect the results of simulations and experiments
    for the evaluation of congestion control mechanisms, and what is
    known about this characteristic in the real world.  We also explain
    why the use of such tools can add considerable power to our
    understanding and evaluation of simulation and testbed scenarios.