[e2e] TCP un-friendly congestion control

Wed Jun 18 09:54:54 PDT 2003

My apologies, I thought I had removed the attachment. I also truncated the message from Sylvain.  The full message is below:

"Hi Les,

There is very little buffer space on Cisco 7600 router. By default the size of the buffer is 40 packets and the maximum value is 4096. (40 packets at 10 Gbps => 0.048ms of queuing delay!!!). Next generation of 10 GE Cisco modules will have 150 ms of buffer space. I think that Juniper router have large buffer spaces too.

Ijong is right; the high performance is due to a large buffer at the bottleneck. In our case (LSR-IPv6), the bottleneck was the end-host and its NIC. That's why we need a large txqueuelen on the end host. (ifconfig eth0 txqueuelen 1000000). In the case of LSR-IPv4, we have limited the TCP buffer size.

Please refer to the presentation attached. On slide #9 and #10 it is explained how tune TCP buffer and txqueuelen.

You will see that large buffer spaces considerably affect queuing delay. The only new stack which doesn't affect delay is FAST. FAST maintains low queuing delay. That's one reason why I am a strong supporter of Steven's work. 

Other stacks (HSTCP, scalable and GridDT) fill the buffer up before loosing a packet and reducing the congestion window. If you have small buffer the end-to-end delay is not too much affected, if you have large buffer you need QoS mechanisms in order to maintain low queuing delay for real time traffic.

The effect on queuing delay is definitely something to take into account in the evaluation of TCP stack.

I am working on a paper which should explain this.

Cheers,

Sylvain"

The web page referred to by Sylvain is at: 

http://sravot.home.cern.ch/sravot/presentation/Grid-DT.ppt

-----Original Message-----
From: Cottrell, Les 
Sent: Wednesday, June 18, 2003 9:46 AM
To: 'Injong Rhee'; 'end2end-interest at postel.org'
Subject: RE: [e2e] TCP un-friendly congestion control

Sylvain Ravot of Caltech, but based at CERN provided me with the following information:

"There is very little buffer space on Cisco 7600 router. By default the size of the buffer is 40 packets and the maximum value is 4096. (40 packets at 10 Gbps => 0.048ms of queuing delay!!!). Next generation of 10 GE Cisco modules will have 150 ms of buffer space. I think that Juniper router have large buffer spaces too.

Ijong is right; the high performance is due to a large buffer at the bottleneck. In our case (LSR-IPv6), the bottleneck was the end-host and its NIC. That's why we need a large txqueuelen on the end host. (ifconfig eth0 txqueuelen 1000000). In the case of LSR-IPv4, we have limited the TCP buffer size.

Please refer to the presentation attached. On slide #9 and #10 it is explained how tune TCP buffer and txqueuelen."

The presentation is NOT attached, but is available at: http://sravot.home.cern.ch/sravot/presentation/Grid-DT.ppt

-----Original Message-----
From: Injong Rhee [mailto:rhee at eos.ncsu.edu] 
Sent: Monday, June 09, 2003 9:27 AM
To: end2end-interest at postel.org
Subject: RE: [e2e] TCP un-friendly congestion control

I came to know this work recently too.  I suspect that the high performance migth be due to a large buffer space some place on DropTail router network. In the droptail router with buffer space K, the congestion window drops down to (B*D + T)/2. If T is as much as B*D, then TCP drops its window down to B*D which is the capacity of the link. If RED is used, the result will be quite different, i believe because packet drop will happen well before the buffer fills up. Also DropTail with a large buffer adds a lot of congestion delays.

 Could somebody confirm the amount of router buffer space at the bottleneck of this path and whether RED is being used? whether the buffer space is pooled or dedicated to a port? At some point, Tom Kelly reported a very little buffer space. But that could have been changed.

-----Original Message-----
From: end2end-interest-admin at postel.org [mailto:end2end-interest-admin at postel.org]On Behalf Of Richard Carlson
Sent: Monday, June 09, 2003 11:04 AM
To: end2end-interest at postel.org
Subject: Re: [e2e] TCP un-friendly congestion control

As chair of the Internet 2 Land Speed Record judging committee, I can confirm that TCP over long fat networks is not hopeless.  The last 2 records, using IPv4 and IPv6, show that hosts can push the network to the limit.  For the latest IPv4 record a team from the US and Europe ran Iperf for 3700 seconds (yes just over 1 hour) between 2 PC's with 10 Gigabit Ethernet NICs.  They achieved a throughput rate of 2.3 Gbps, with the connection being limited by a transatlantic OC-48 link.  The latest IPv6 record is being processed now, but it is equally impressive.  See the I2-LSR web page http://lsr.internet2.net for details.

Another thing to consider, Les Cottrell from SLAC ran some tests with these 10 Gig NICs and tested the standard Linux 2.4 TCP, FAST TCP, HPTCP, and one other high performance implementation.  These tests showed no difference between any of the TCP implementations over 1 particular network path.  If Les is on this list maybe he can provide a pointer to these results.

Rich

At 05:26 PM 6/6/03 -0400, Craig Partridge wrote:

>In message <20030606140028.A59651 at xorpc.icir.org>, Luigi Rizzo writes:
>
> >exactly -- that's the whole point, the packet rate is so high that
> >there are many chances to have occasional losses anywhere in the 
> >system, not just due to the channel but even because of operating 
> >system issues on some of the intermediate boxes or end nodes.
>
>OK, let me get on my high horse here for a moment.
>
>The original poster asserted that in an environment where the network
>went at 1 Gbps and had 50ms of delay, TCP was hopeless.
>
>The point I was trying to drive home is that it is not hopeless.  That
>you have to define the environment far more carefully before you assert 
>that TCP can or cannot do the job.  One of my frustrations these days 
>is people who fail to be careful.  I was trying to encourage care in 
>the problem statement.
>
>Thanks!
>
>Craig

------------------------------------

Richard A. Carlson                              e-mail: RACarlson at anl.gov
Network Research Section                        phone:  (630) 252-7289
Argonne National Laboratory                     fax:    (630) 252-4021
9700 Cass Ave. S.
Argonne,  IL 60439

---
Incoming mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.488 / Virus Database: 287 - Release Date: 6/5/2003

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.488 / Virus Database: 287 - Release Date: 6/5/2003