[Tsvwg] Re: [tcpm] Re: [e2e] Are you interested in TOEs and related issues(Resend)

[Sunay Tripathi]

Rick,

> > Yes, that was one of the first thoughts that came to my mind when I
> > started looking at this space. Its not just security issues but bugs and
> > new features. TCP despite being mature still sees couple of RFC or tweaks
> > a year that needs to be rolled out to existing systems. Apparantly the
> > TOE vendors had thought about that. The firmware running the protocol code
> > actually lives on the host and is delivered via exisiting delivery
> > mechanism (packages in Solaris which can be upgraded). When machine boots
> > and interface is configured, the protocol firmware is sucked in. So
> > delivering fix or new revs in pretty similar to exisiting mechanisms.
> > Also, the source code is still 'C' based so people like me can still
> > work with it (just need to use some magic in the end to create the
> > firmware instead of complier). Some vendors we talked to have promised
> > that they can pretty much suck in the same code that runs on the host
> > into the card (as long as we can clean some complier specific directives
> > etc). Although I must admit that this aspect (common source files) of TOE
> > has not been investigated by us at all so far.
> 
> That seems like a path out when the TOE is based on an embedded CPU, but doesn't
> seem like it would work for those TOEs that are ASICs.

The TOEs I am talking about have 1 or more embedded ARM processor (seems to 
be the more common model these days). 

> And a mostly rhetorical question - strictly in the context of TCP comms and not
> RDMA and the like - if one can code tightly enough for an embedded CPU a TCP/IP
> stack including comms with the host across the I/O bus with enough performance
> to be reasonable for a given link, can't one presumeably make the host's TCP/IP
> stack similarly efficient?  If nothing else that seems to be alluded to by the
> last sentence in the quoted paragraph.

I think Craig did ask this question before and the benefit is not so much
from the protocol processing saving but from the data movement. Below is
a portion of my earlier reply to Craig's email.

"
1) Extra processor(s) buried in the TOE for networking processing which is
   hidden from the kernel and leaves the host CPU to do more application
   related work. Saves the cost of licences for application which take
   number of CPU into account (oracle is one such application cited).
2) On low end (1-2 CPU) x86 based machines, cost of adding a processor
   is much higher than adding a TOE (I personally haven't verified this).
3) For the up and coming 10Gb NICs, TOE will help saturate the link. Some
   vendors assert that TOE will be required to support 10Gb NICs.
4) Performance reasons. Just the LSO aspect of TOE (sending large chunks of
   data and letting the TOE split it up in mss size pieces) and ack
   coalescing gives a pretty good boost (our own prototypes indicates that
   this is true). The gains are by optimizing data movement and not by
   offloading protocol processing.
5) TOE is necessary for RDMA, iSCSI etc. for layering reasons. I am not
   involved with RDMA so someone who is an expert can probably comment on
   this part.
6) TOE based NIC are already making pretty good headway in embedded space.
   The technology is already maturing so why not use it in broader market.
"

Cheers,
Sunay

> 
> rick jones
> -- 
> Wisdom Teeth are impacted, people are affected by the effects of events.
> these opinions are mine, all mine; HP might not want them anyway... :)
> feel free to post, OR email to raj in cup.hp.com  but NOT BOTH...
> 


-- 
Sunay Tripathi
Senior Staff Engineer,
Solaris Kernel Networking,
Sun MicroSystems Inc.

email: sunay at eng.sun.com		 Phone:	650-786-6007 (W)