[e2e] Short Fat Networks, tcp and Policers

[a simpleton]

Dear e2e list,
My employer ( who shall remain nameless) has, as part of its traffic
enforcement module, a "policer" block, which marks and discards, a'la
rfc2698 and MEF, L2 frames based on bandwidth and burst. Basically -
too many frames that are sent beyond a given rate are discarded. 

As is well  known, tcp does not respond well to this kind of treatment, 
although you can find in cisco/juniper manuals various recomendations
how to configure the policer parameters to improve performance. I was
given the task to try and figure out what could be done to improve the
situation. What I'd like to do here is scribble out a rather informal 
sketch of how tcp handles policers and what I think the main sticking
 points are. 
 
A) Minimum throughput
=======================
 
When you browse through tcp literature, you  will usually come across
 the Bandwidth x Delay=Window formula. It has a very straightforward 
meaning: If you want to get a full utilization of the available bandwidth,
 you better have a window that is large enough to fill the RTT with packets.
 
Even a simpleton like myself knows this. But let us ask the question in 
reverse: How *low* a bandwidth can we pass through a line with a given, fixed,
 delay? Some simple thinking will show you than it is (almost) impossible to 
deliver less that a  few MSS each RTT. Why? Because even during slow start, tcp
sends a new packet once an ack is recieved.
 
What is the implication of the above? That in order to cause a tcp flow to use *less* than the
physically available link, one must *increase*  the RTT. Of course, this is exactly what shapers
do, by buffering the data. alternatively one could buffer the acks ( and I think that "ack holding"
schemes have been proposed in the literature), but that requires L3- L4 knowledge and treatment of 
piggybACKs.
 
B) tcp timeout Phase
========================

So what happens when you try to limit bandwidth below the MSS/RTT limit? A policer achieves this
by discarding all packets that are non conforming, and this will cause the session to run in a
burst-timeout-burst phase. Typical implementations set timeouts to as much as several hundred msec.
In this phase ( that is , when ther required bandwidth is less that Const X MSS/RTT), the bandwidth 
can range from zero to [Policer Burst Size]/[Tcp Timeout] . This phase is not efficient, and causes huge
bursts in the network. For timouts around 500 msec, a 4 mb/s session ( which is not unreasonal on LANs
and WANs) needs a policer burst of 512KB. since bursts tend to get translated into buffer sizes, we see
that a single service eats up quite a large buffer size. The irony being, that in the timeout phase, the
burst is not buffered, because if the burst was buffered, the RTT would increase, and we wouldn't have been
in the Timoeout phase in the first place.

C) Partial window phase
========================

A reasonable way to generate sub-physical line rates, ( without adding to the RTT), is to cause 
the tcp to work at a window that is less that [physical Rate]x[RTT]. The frame pattern would be
something like M frames every RTT, with MxMSS/RTT ~ [policed rate]. This is a much better behaviour
than the timeout phase, and a good policer design should strive to reach this phase. As pointed out
before, this phase cannot exist when the required rate is too low, or the RTT is too short.

D) Stability of the partial window phase
========================================

Eventually, because of slow start or congestion avoidence, the number of frames in the window M
slowly creeps up, until the policer "realizes" that the policed rate has been passed, and the policer
will discard a series of frames. If this is done delicately enough ( suppose using a RED like algorithm),
Fast retransmission will take place and the session shall be able to slow start its way back to the target of 
M frames per RTT. 

If the policing is too drastic, either an entire window of M packets will be discarded, or the fast-retransmit frame itself will be lost, and a timeout will occur. 
E) tcp defence lines and policers
==================================
tcp has three defence lines against congestion
* self clocking
* congestion window + slow start
* retransmission timeout
Not only do they protect the network, they also control the bandwidth that the application recieves. 
Policers neutralize completely the first line of defence, since they have no effect on the RTT ( or on the
more subtle inter packet gap ). The only way that policers can indicate rate to the tcp layer is by
packet discard. Tcp responds to packet discard by retransmission timeouts or fast retransmission, both
are considered inefficient, but compared to the huge problems caused by timeouts, the slight ineffciency caused by fast retransmission induced slow start, is minor. The estimation of timeouts based on averaged RTT
statistics are totally irrelevant when the actual network performance is below 100msec, but the recommended
tcp tick  is 500msec.

F) What is the point?!
======================
The points are 
1- policers are much easier to build that shapers, so we should start to understand them.
2- acceptance tests are usually done with very short RTT times, so that the timeout phase is
quite relevant.
3- All these exponential smoothings and estimations of RTT RMS are useless if the smallest timeout
is 300msec, and typical LAN's are less that 50msec.
4- alot of TCP improvements have been based around "LFN"'s but it may turn out that alot of the broad band networks are really "SFN"'s (Short Fat Networks). 
G) The End
============
Thank you for you patience, comments are welcome.

Yet Another Simpleton

		
---------------------------------
Do you Yahoo!?
Win a $20,000 Career Makeover at Yahoo! HotJobs 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.postel.org/pipermail/end2end-interest/attachments/20040510/38748844/attachment.html