[e2e] Are you interested in TOEs and related issues

[Sunay Tripathi]

Craig,

> Well, the interesting thing is we went down this path in the early 1980s
> and found that TOEs didn't work.
> 
> If I try to distill all that we learned in the 1980s into one question, I
> come out with:
> 
>     In the 1980s we discovered that communicating with a TOE over a bus
>     about a TCP connection was as expensive or more expensive than
>     simply handling the TCP connection in the main processor.  What about
>     today's TOE designs makes them different?

I don't have too much history (1980s predates my career) but there were
some attempts in mid 90s to do the same and compared to that, the
following reasons seem to be the driving force:

1) Extra processor(s) buried in the TOE for networking processing which is
   hidden from the kernel and leaves the host CPU to do more application
   related work. Saves the cost of licences for application which take
   number of CPU into account (oracle is one such application cited).
2) On low end (1-2 CPU) x86 based machines, cost of adding a processor
   is much higher than adding a TOE (I personally haven't verified this).
3) For the up and coming 10Gb NICs, TOE will help saturate the link. Some
   vendors assert that TOE will be required to support 10Gb NICs.
4) Performance reasons. Just the LSO aspect of TOE (sending large chunks of
   data and letting the TOE split it up in mss size pieces) and ack
   coalescing gives a pretty good boost (our own prototypes indicates that
   this is true). The gains are by optimizing data movement and not by
   offloading protocol processing.
5) TOE is necessary for RDMA, iSCSI etc. for layering reasons. I am not
   involved with RDMA so someone who is an expert can probably comment on
   this part.
6) TOE based NIC are already making pretty good headway in embedded space.
   The technology is already maturing so why not use it in broader market.

Note that the above claims are in no particular order of importance and made
my TOE vendors in general. Of these, I personally do agree with 1 and 4 but
that iteself doesn't mean that TOE will make it in general purpose networking.

It would be interesting to see if you and others in the list agree or
disagree with these claims.

Thanks,
Sunay

-- 

Sunay Tripathi
Senior Staff Engineer,
Solaris Kernel Networking,
Sun MicroSystems Inc.

email: sunay at eng.sun.com		 Phone:	650-786-6007 (W)