[e2e] Are you interested in TOEs and related issues

[Sunay Tripathi]

> 
> In message <200403040455.i244twSG699301 at jurassic.eng.sun.com>, Sunay Tripathi w
> rites:
> 
> >I don't have too much history (1980s predates my career) but there were
> >some attempts in mid 90s to do the same and compared to that, the
> >following reasons seem to be the driving force:
> >
> >1) Extra processor(s) buried in the TOE for networking processing which is
> >   hidden from the kernel and leaves the host CPU to do more application
> >   related work. Saves the cost of licences for application which take
> >   number of CPU into account (oracle is one such application cited).
> 
> Right, and the problem is that if the TOE is very sophisticated, you can't
> hide it from the host CPU.  Remember when the application says "write",
> on a socket, some magic has to occur to move a buffer from application space
> to the right TCP connection -- and past experience says that magic has
> 90% of your non-memory moving costs...

The question is whether you want to hide the TOE from the application or
kernel as well? The current implementations override the S/W stack and 
put their own hooks in sockets so that TOE is kind of hidden from application
as well as the kernel (or atleast the kernel can do nothing about it).
I am pretty sure that this won't fly. You can hide the TOE from the
application but the kernel needs to control it. And once the kernel
controls the TOE, the magic becomes easy (or not expensive atleast).
My current estimate is that I probably spend 10% more instructions in
connection setup/teardown but in return I have an extra processing
element. And to sweeten the deal, I am moving data in bigger chunks
and saving on the memory costs also. 
 

> >3) For the up and coming 10Gb NICs, TOE will help saturate the link. Some
> >   vendors assert that TOE will be required to support 10Gb NICs.
> 
> For any speed X, where X is viewed as large, there will be people claiming
> they have just the innovation needed to achieve that speed.

Agreed totally :)

Thanks,
Sunay

> >4) Performance reasons. Just the LSO aspect of TOE (sending large chunks of
> >   data and letting the TOE split it up in mss size pieces) and ack
> >   coalescing gives a pretty good boost (our own prototypes indicates that
> >   this is true). The gains are by optimizing data movement and not by
> >   offloading protocol processing.
> 
> Agreed -- plenty of evidence that if you don't do protocol processing but
> just help moving memory, you win.
> 
> >5) TOE is necessary for RDMA, iSCSI etc. for layering reasons. I am not
> >   involved with RDMA so someone who is an expert can probably comment on
> >   this part.
> 
> Certainly for memory moving it is.
> 
> >6) TOE based NIC are already making pretty good headway in embedded space.
> >   The technology is already maturing so why not use it in broader market.
> 
> In the embedded space you've got a different cost-benefit tradeoff.  The
> embedded processor may often have a single application...
> 
> Craig
> 


-- 
Sunay Tripathi
Senior Staff Engineer,
Solaris Kernel Networking,
Sun MicroSystems Inc.

email: sunay at eng.sun.com		 Phone:	650-786-6007 (W)