<DIV>Dear e2e list,</DIV>
<DIV>My employer ( who shall remain nameless) has, as part of its traffic<BR>enforcement module, a "policer" block, which marks and discards, a'la<BR>rfc2698 and MEF, L2 frames based on bandwidth and burst. Basically -<BR>too many frames that are sent beyond a given rate are discarded. </DIV>
<DIV><BR>As is well known, tcp does not respond well to this kind of treatment, <BR>although you can find in cisco/juniper manuals various recomendations<BR>how to configure the policer parameters to improve performance. I was<BR>given the task to try and figure out what could be done to improve the<BR>situation. What I'd like to do here is scribble out a rather informal <BR>sketch of how tcp handles policers and what I think the main sticking<BR> points are. </DIV>
<DIV> </DIV>
<DIV>A) Minimum throughput<BR>=======================</DIV>
<DIV> </DIV>
<DIV>When you browse through tcp literature, you will usually come across<BR> the Bandwidth x Delay=Window formula. It has a very straightforward <BR>meaning: If you want to get a full utilization of the available bandwidth,<BR> you better have a window that is large enough to fill the RTT with packets.</DIV>
<DIV> </DIV>
<DIV>Even a simpleton like myself knows this. But let us ask the question in <BR>reverse: How *low* a bandwidth can we pass through a line with a given, fixed,<BR> delay? Some simple thinking will show you than it is (almost) impossible to <BR>deliver less that a few MSS each RTT. Why? Because even during slow start, tcp<BR>sends a new packet once an ack is recieved.</DIV>
<DIV> </DIV>
<DIV>What is the implication of the above? That in order to cause a tcp flow to use *less* than the<BR>physically available link, one must *increase* the RTT. Of course, this is exactly what shapers<BR>do, by buffering the data. alternatively one could buffer the acks ( and I think that "ack holding"<BR>schemes have been proposed in the literature), but that requires L3- L4 knowledge and treatment of <BR>piggybACKs.</DIV>
<DIV> </DIV>
<DIV>B) tcp timeout Phase<BR>========================</DIV>
<DIV><BR>So what happens when you try to limit bandwidth below the MSS/RTT limit? A policer achieves this<BR>by discarding all packets that are non conforming, and this will cause the session to run in a<BR>burst-timeout-burst phase. Typical implementations set timeouts to as much as several hundred msec.<BR>In this phase ( that is , when ther required bandwidth is less that Const X MSS/RTT), the bandwidth <BR>can range from zero to [Policer Burst Size]/[Tcp Timeout] . This phase is not efficient, and causes huge<BR>bursts in the network. For timouts around 500 msec, a 4 mb/s session ( which is not unreasonal on LANs<BR>and WANs) needs a policer burst of 512KB. since bursts tend to get translated into buffer sizes, we see<BR>that a single service eats up quite a large buffer size. The irony being, that in the timeout phase, the<BR>burst is not buffered, because if the burst was buffered, the RTT would increase, and we wouldn't have been<BR>in the Timoeout phase in the first
place.</DIV>
<DIV><BR>C) Partial window phase<BR>========================</DIV>
<DIV><BR>A reasonable way to generate sub-physical line rates, ( without adding to the RTT), is to cause <BR>the tcp to work at a window that is less that [physical Rate]x[RTT]. The frame pattern would be<BR>something like M frames every RTT, with MxMSS/RTT ~ [policed rate]. This is a much better behaviour<BR>than the timeout phase, and a good policer design should strive to reach this phase. As pointed out<BR>before, this phase cannot exist when the required rate is too low, or the RTT is too short.</DIV>
<DIV><BR>D) Stability of the partial window phase<BR>========================================</DIV>
<DIV><BR>Eventually, because of slow start or congestion avoidence, the number of frames in the window M<BR>slowly creeps up, until the policer "realizes" that the policed rate has been passed, and the policer<BR>will discard a series of frames. If this is done delicately enough ( suppose using a RED like algorithm),<BR>Fast retransmission will take place and the session shall be able to slow start its way back to the target of <BR>M frames per RTT. </DIV>
<DIV><BR>If the policing is too drastic, either an entire window of M packets will be discarded, or the fast-retransmit frame itself will be lost, and a timeout will occur. </DIV>
<DIV>E) tcp defence lines and policers<BR>==================================</DIV>
<DIV>tcp has three defence lines against congestion<BR>* self clocking<BR>* congestion window + slow start<BR>* retransmission timeout</DIV>
<DIV>Not only do they protect the network, they also control the bandwidth that the application recieves. <BR>Policers neutralize completely the first line of defence, since they have no effect on the RTT ( or on the<BR>more subtle inter packet gap ). The only way that policers can indicate rate to the tcp layer is by<BR>packet discard. Tcp responds to packet discard by retransmission timeouts or fast retransmission, both<BR>are considered inefficient, but compared to the huge problems caused by timeouts, the slight ineffciency caused by fast retransmission induced slow start, is minor. The estimation of timeouts based on averaged RTT<BR>statistics are totally irrelevant when the actual network performance is below 100msec, but the recommended<BR>tcp tick is 500msec.</DIV>
<DIV><BR>F) What is the point?!<BR>======================</DIV>
<DIV>The points are </DIV>
<DIV>1- policers are much easier to build that shapers, so we should start to understand them.</DIV>
<DIV>2- acceptance tests are usually done with very short RTT times, so that the timeout phase is<BR>quite relevant.</DIV>
<DIV>3- All these exponential smoothings and estimations of RTT RMS are useless if the smallest timeout<BR>is 300msec, and typical LAN's are less that 50msec.</DIV>
<DIV>4- alot of TCP improvements have been based around "LFN"'s but it may turn out that alot of the broad band networks are really "SFN"'s (Short Fat Networks). </DIV>
<DIV>G) The End<BR>============</DIV>
<DIV>Thank you for you patience, comments are welcome.</DIV>
<DIV><BR>Yet Another Simpleton</DIV><p>
<hr size=1><font face=arial size=-1>Do you Yahoo!?<br><a href="http://pa.yahoo.com/*http://us.rd.yahoo.com/hotjobs/hotjobs_mail_signature_footer_textlink/evt=23983/*http://hotjobs.sweepstakes.yahoo.com/careermakeover">Win a $20,000 Career Makeover at Yahoo! HotJobs </a>