[e2e] TCP "experiments"

[Lachlan Andrew]

Greetings John,

On 28 July 2013 11:25, John Day <jeanjour at comcast.net> wrote:
> At 10:33 AM +1000 7/28/13, Lachlan Andrew wrote:
>>
>> You are absolutely right that testbed experiments should be performed
>> before "live" experiments.  However, it is not so much the size of the
>> network as the mix of applications running on it that makes the test
>> representative.
>
> Are you telling me that we don't have good statistical models for the
> behavior of large numbers of users?   I would suggest that this is just
> laziness, irresponsible behavior or both.

It depends on what you mean by "good".  We have lots of models of lots
of aspects of their behaviour, but cannot possibly hope to model all
of them.
If you go to TCPM and say "Here is an enhancement of TCP.  I know it
works because I have a statistical model of it", do you think they
will be happy, however "good" your model is?  I don't.

There is a need for testing at many levels.  We need to test with
simulations / models, with testbeds with synthetic users, with
testbeds with real users, and with the real internet.  Otherwise, we
don't understand how things will really behave.

>> Of course, that doesn't excuse un-monitored deployments as occurred
>> when Linux started using BIC as the default.  To my mind, the solution
>> would be for the IETF to provide more practical guidance on how to
>> perform limited-scale, monitored tests on the real Internet.
>
> That is not the job of an organization dedicated to the maintenance of a
> production network.

True.  That is why I don't suggest that NANOG provide such a list.  I
see the role of the IETF as to promote and guide innovation in the
Internet, rather than to maintain it.

> Have you ever seen what researchers in other fields do to ensure that they
> get accurate and reliable results?  Triple distillations, putting precisely
> the same amount in 2000 test tubes, making your own reagents?

In physics and chemistry, experiments are done on the real physical
and chemical world.  In medicine, they run clinical trials on real
people.

As I said, I'm not proposing that people should run tests on the
public internet *instead* of doing testbed studies.  I'm saying that
there comes a time when things need to be tested in reality.  The IETF
has an opportunity to guide these tests.  If it chooses not to guide
them, then they will be carried out without its guidance.

> I would question whether it is the job of *researchers* to get a protocol
> approved by the IETF for use in product.

I thought that was the role of experimental RFCs.  Who will do the
experiments, if not researchers?

Also, I was explicitly not talking about putting things in products.
I was talking about doing experiments on the Internet.  I believe that
Cubic should not have been deployed as the Linux default until it had
been tested properly.  I maintain that "proper testing" includes
properly monitored experiments on the Internet.

The issue of what "experiments" are allowed on the Internet goes to
the heart of things like the ACM's IMC.  Should CAIDA not be allowed
to experiment on the Internet because it is a production network?
Should companies like Akamai be allowed to repeat the (sometimes quite
burdensome) measurements on a regular basis?  How often?  The IETF is
welcome not to care about such things, but I was suggested that they
may have useful things to say.  If they don't care about such things,
why should they care about other experiments?  Since so few TCP flows
are affected by the congestion control algorithm, why is it so
sacrosanct?

>>  The IETF will be
>> most relevant if its processes reflect its power.
>
> But you do have a good point.  I have some interesting ideas for more
> efficient use of the power grid, I think I will deploy them to see how they
> affect the balance of the grid.  Gee, what could go wrong!

The makers of electric vehicles are doing exactly that.  Of course,
EVs have to be approved for electrical safety, but that doesn't do
anything to show what effect they have on the balance of the grid.
EVs are a nightmare.

(As an aside, last week, at last week's IEEE Power and Energy Society
General Meeting we were told that in Alberta, the network operator has
no say in what new generation is placed where.  They just have to deal
with what happens.  The situation is not ideal, but is not unique to
the Internet.  I assume you were being sarcastic, but if you really do
have some ideas for more efficient energy use, I certainly hope that
you pursue them.  That is much more likely to help the world than any
development in cyberspace ever could.)

Cheers,
Lachlan

-- 
Lachlan Andrew  Centre for Advanced Internet Architectures (CAIA)
Swinburne University of Technology, Melbourne, Australia
<http://caia.swin.edu.au/cv/landrew>
Ph +61 3 9214 4837