[Tsvwg] Re: [e2e] Are you interested in TOEs and related issues

[Sunay Tripathi]

<SNIP>
> > 3) For the up and coming 10Gb NICs, TOE will help saturate 
> > the link. Some
> >    vendors assert that TOE will be required to support 10Gb NICs.
> 
> 
> TOE will bring CPU utilization down of course, but 10GbE NIC will
> saturate the link with or without a TOE - much like GbE NICs do it now.

I think the TOE vendors meant that single CPU won't be able to saturate
the 10Gb NIC on a more meaningful workloads than just a simple throughput
tests. Even a 2 CPU machine which wants to do some work also might also
get constrained.

> > 4) Performance reasons. Just the LSO aspect of TOE (sending 
> > large chunks of
> >    data and letting the TOE split it up in mss size pieces) and ack
> >    coalescing gives a pretty good boost (our own prototypes 
> > indicates that
> >    this is true). 
> 
> Just curious - in your testing, how much CPU is freed up by the ack
> coalescing?

We actually always try to run the CPUs to saturation (keep increasing
the load) and we can get a double digit percentage boost on a web
like workload (multiple simultaneous connections). I didn't verify
exactly where the gains were coming from but my feeling was that
more efficient data movement and better locality was contributing to
bulk of the gains.

> >The gains are by optimizing data movement and not by
> >    offloading protocol processing.
> 
> LSO is a 'stateless' offload (much like checksum offload) so one doesn't
> need a TOE to support it; 

It depends. If the kernel does a write based on open window, then yes
you are right. But if you have full TOE and you pass large chunks of
data to TOE (does wonder when sendfile/sendfilev is in play) and let
TOE figure out when to send it, the gains increase. In the later case,
TOE needs teh TCP state.

> most modern NICs (including our 10GbE Adapter) do LSO. Also, for normal
> frames LSO has quite dramatic effect at 10GbE rates, but for Jumbo
> frames performance gain diminishes to ~7%.

Do you have any insight on why you would see a degradation with Jumbo
frames? As long as you are stateless, the jumbograms should show
no degradation. 

Thanks,
Sunay


> 
> 
> > 5) TOE is necessary for RDMA, iSCSI etc. for layering 
> > reasons. I am not
> >    involved with RDMA so someone who is an expert can 
> > probably comment on
> >    this part.
> > 6) TOE based NIC are already making pretty good headway in 
> > embedded space.
> >    The technology is already maturing so why not use it in 
> > broader market.
> > 
> > Note that the above claims are in no particular order of 
> > importance and made my TOE vendors in general. Of these, I 
> > personally do agree with 1 and 4 but that iteself doesn't 
> > mean that TOE will make it in general purpose networking.
> > 
> > It would be interesting to see if you and others in the list 
> > agree or disagree with these claims.
> > 
> > Thanks,
> > Sunay
> > 
> > -- 
> > 
> > Sunay Tripathi
> > Senior Staff Engineer,
> > Solaris Kernel Networking,
> > Sun MicroSystems Inc.
> > 
> > email: sunay at eng.sun.com		 Phone:	650-786-6007 (W)
> > 
> > 
> > 
> > 
> 
> 
> Regards, 
> Leonid Grossman
> www.s2io.com
> 
> 
> > _______________________________________________
> > tsvwg mailing list
> > tsvwg at ietf.org
> > https://www1.ietf.org/mailman/listinfo/tsvwg
> > 
> 


-- 
Sunay Tripathi
Solaris Kernel Networking,
Sun MicroSystems Inc.

email: sunay at eng.sun.com		 Phone:	650-786-6007 (W)