[tcpm] Re: [e2e] Are you interested in TOEs and related issues (Resend)

[Sunay Tripathi]

[Sorry I was out travelling and in the meanwhile this thread has
morphed into router design :) Anyway, I hope someone can start a
new thread with the router design subject while we continue on
TOE discussion - Thanks]

David,

> I wouldn't say in general that we found out that TOEs didn't work.  I 
> think a more qualified comment of what we found out was that in an 
> environment where the network is the bottle neck, TOEs did not provide 
> any cost effective benefits; and as CPUs got faster, the TOEs just 
> provided costs without benefits. :-)
> 
> TOE has become a buzz-word that needs to be qualified.  Some folks 
> include Checksum Offload and Large Segmentation Offload when talking 
> about TOE.  When others talk about TOE, they refer to offloading the 
> whole TCP/IP stack.

I think we are talking about offloading the whole stack or to the point
where the NIC needs to keep the state. The stateless offload (cksum
offload and LSO has already been happening for a while).

> With 10GigE, we are in the realm where the network is not the bottle 
> neck.  Even with GigE, the CPU has to do significant work to keep the 
> card fed, leaving little time for doing any application work.  I think 
> there are very few if any CPUs available today that can keep a 10GigE 
> pipe fed, much less handle an inbound 10GigE stream.

Our experience as well. The fastest intel or sparc CPU can saturate the
1GigE but there is not much room left for doing meaningful work by the
application.

> Features such as Checksum Offload and LSO are features that have low 
> impact on the OS and its TCP/IP stack.  Other features such as having a 
> timer to delay and aggregate transmit interrupts also help to reduce 
> the amount of work that the CPU has to do to process the data.
> 
> For me, the question is what can be done in a 10GigE NIC to allow the 
> host CPU to handle both sending and receiving full speed on that 
> interface?  TOE is just one part of that discussion and the danger is 
> that if too much focus is put on TOE, the other aspects of how to 
> improve how the CPU and the NIC communicate will not be properly 
> investigated.

You defintely have to fanout to distribute load across multiple CPUs if
you don't have TOE and CPU speed doesn't scale dramatically in couple of years.
We are developing a dynamic polling/interrupt mechanism to use general
purpose CPU to saturate 10GigE. Using TOE based cards is another possible
approach (you still need to deal with moving a giga byte of data across
your back plane though).

Cheers,
Sunay


> 
> 			-David Borman
> 
> On Mar 3, 2004, at 12:07 AM, Craig Partridge wrote:
> 
> >
> > Well, the interesting thing is we went down this path in the early 
> > 1980s
> > and found that TOEs didn't work.
> >
> > If I try to distill all that we learned in the 1980s into one 
> > question, I
> > come out with:
> >
> >     In the 1980s we discovered that communicating with a TOE over a bus
> >     about a TCP connection was as expensive or more expensive than
> >     simply handling the TCP connection in the main processor.  What 
> > about
> >     today's TOE designs makes them different?
> >
> > Craig
> >
> > _______________________________________________
> > tcpm mailing list
> > tcpm at ietf.org
> > https://www1.ietf.org/mailman/listinfo/tcpm
> 


-- 
Sunay Tripathi
Senior Staff Engineer,
Solaris Kernel Networking,
Sun MicroSystems Inc.

email: sunay at eng.sun.com		 Phone:	650-786-6007 (W)