[e2e] Are you interested in TOEs and related issues

[Craig Partridge]

In message <200403040455.i244twSG699301 at jurassic.eng.sun.com>, Sunay Tripathi w
rites:

>I don't have too much history (1980s predates my career) but there were
>some attempts in mid 90s to do the same and compared to that, the
>following reasons seem to be the driving force:
>
>1) Extra processor(s) buried in the TOE for networking processing which is
>   hidden from the kernel and leaves the host CPU to do more application
>   related work. Saves the cost of licences for application which take
>   number of CPU into account (oracle is one such application cited).

Right, and the problem is that if the TOE is very sophisticated, you can't
hide it from the host CPU.  Remember when the application says "write",
on a socket, some magic has to occur to move a buffer from application space
to the right TCP connection -- and past experience says that magic has
90% of your non-memory moving costs...

>3) For the up and coming 10Gb NICs, TOE will help saturate the link. Some
>   vendors assert that TOE will be required to support 10Gb NICs.

For any speed X, where X is viewed as large, there will be people claiming
they have just the innovation needed to achieve that speed.

>4) Performance reasons. Just the LSO aspect of TOE (sending large chunks of
>   data and letting the TOE split it up in mss size pieces) and ack
>   coalescing gives a pretty good boost (our own prototypes indicates that
>   this is true). The gains are by optimizing data movement and not by
>   offloading protocol processing.

Agreed -- plenty of evidence that if you don't do protocol processing but
just help moving memory, you win.

>5) TOE is necessary for RDMA, iSCSI etc. for layering reasons. I am not
>   involved with RDMA so someone who is an expert can probably comment on
>   this part.

Certainly for memory moving it is.

>6) TOE based NIC are already making pretty good headway in embedded space.
>   The technology is already maturing so why not use it in broader market.

In the embedded space you've got a different cost-benefit tradeoff.  The
embedded processor may often have a single application...

Craig