[e2e] TCP "experiments"

[John Day]

At 2:18 PM +1000 7/28/13, Lachlan Andrew wrote:
>Greetings John,
>
>On 28 July 2013 11:25, John Day <jeanjour at comcast.net> wrote:
>>  At 10:33 AM +1000 7/28/13, Lachlan Andrew wrote:
>>>
>>>  You are absolutely right that testbed experiments should be performed
>>>  before "live" experiments.  However, it is not so much the size of the
>>>  network as the mix of applications running on it that makes the test
>>>  representative.
>>
>>  Are you telling me that we don't have good statistical models for the
>>  behavior of large numbers of users?   I would suggest that this is just
>>  laziness, irresponsible behavior or both.
>
>It depends on what you mean by "good".  We have lots of models of lots
>of aspects of their behaviour, but cannot possibly hope to model all
>of them.

Law of Large Numbers.  You are splitting hairs for the sake of 
argument.  But then I have been told that all of you believe that 
traffic is self-similar, so it always has the same statistical 
behavior.

>If you go to TCPM and say "Here is an enhancement of TCP.  I know it
>works because I have a statistical model of it", do you think they
>will be happy, however "good" your model is?  I don't.
>
>There is a need for testing at many levels.  We need to test with
>simulations / models, with testbeds with synthetic users, with
>testbeds with real users, and with the real internet.  Otherwise, we
>don't understand how things will really behave.

Precisely my point.  And most of these people are skipping many of those.

>
>>>  Of course, that doesn't excuse un-monitored deployments as occurred
>>>  when Linux started using BIC as the default.  To my mind, the solution
>>>  would be for the IETF to provide more practical guidance on how to
>>>  perform limited-scale, monitored tests on the real Internet.
>>
>>  That is not the job of an organization dedicated to the maintenance of a
>>  production network.
>
>True.  That is why I don't suggest that NANOG provide such a list.  I
>see the role of the IETF as to promote and guide innovation in the
>Internet, rather than to maintain it.
>
>>  Have you ever seen what researchers in other fields do to ensure that they
>>  get accurate and reliable results?  Triple distillations, putting precisely
>>  the same amount in 2000 test tubes, making your own reagents?
>
>In physics and chemistry, experiments are done on the real physical
>and chemical world.  In medicine, they run clinical trials on real
>people.
>
>As I said, I'm not proposing that people should run tests on the
>public internet *instead* of doing testbed studies.  I'm saying that
>there comes a time when things need to be tested in reality.  The IETF
>has an opportunity to guide these tests.  If it chooses not to guide
>them, then they will be carried out without its guidance.

To take your point, there indeed comes a time when the community 
reviewing the test results should decide that the tests warrant in 
vivo trials.

>
>>  I would question whether it is the job of *researchers* to get a protocol
>>  approved by the IETF for use in product.
>
>I thought that was the role of experimental RFCs.  Who will do the
>experiments, if not researchers?

There is a difference between doing research and the process "to get 
a protocol approved by the IETF."  One is scientific, the other has a 
definite self-serving purpose.  I am not saying that is bad, but 
there is a conflict of interest between doing research and getting 
approval.

>
>Also, I was explicitly not talking about putting things in products.
>I was talking about doing experiments on the Internet.  I believe that
>Cubic should not have been deployed as the Linux default until it had
>been tested properly.  I maintain that "proper testing" includes
>properly monitored experiments on the Internet.
>
>The issue of what "experiments" are allowed on the Internet goes to
>the heart of things like the ACM's IMC.  Should CAIDA not be allowed
>to experiment on the Internet because it is a production network?

Should EPRI be allowed to experiment with the electric grid that 
comes to my house?

Should CAIDA or Akamai be running some kind of stress test that just 
happens to affect traffic down the road as a remote surgery commences?

>Should companies like Akamai be allowed to repeat the (sometimes quite
>burdensome) measurements on a regular basis?  How often?  The IETF is
>welcome not to care about such things, but I was suggested that they
>may have useful things to say.  If they don't care about such things,
>why should they care about other experiments?  Since so few TCP flows
>are affected by the congestion control algorithm, why is it so
>sacrosanct?

The last thing anything should be should be sacrosanct.  However, the 
distinction remains between production and research.

>
>>>   The IETF will be
>>>  most relevant if its processes reflect its power.
>>
>>  But you do have a good point.  I have some interesting ideas for more
>>  efficient use of the power grid, I think I will deploy them to see how they
>>  affect the balance of the grid.  Gee, what could go wrong!
>
>The makers of electric vehicles are doing exactly that.  Of course,
>EVs have to be approved for electrical safety, but that doesn't do
>anything to show what effect they have on the balance of the grid.
>EVs are a nightmare.

Does make air-conditioning during a heat-wave look like an easy 
problem, doesn't it?  ;-)

>
>(As an aside, last week, at last week's IEEE Power and Energy Society
>General Meeting we were told that in Alberta, the network operator has
>no say in what new generation is placed where.  They just have to deal
>with what happens.  The situation is not ideal, but is not unique to
>the Internet.  I assume you were being sarcastic, but if you really do
>have some ideas for more efficient energy use, I certainly hope that
>you pursue them.  That is much more likely to help the world than any
>development in cyberspace ever could.)

We can leave that for another time.

Take care,
John

>
>Cheers,
>Lachlan
>
>--
>Lachlan Andrew  Centre for Advanced Internet Architectures (CAIA)
>Swinburne University of Technology, Melbourne, Australia
><http://caia.swin.edu.au/cv/landrew>
>Ph +61 3 9214 4837