[e2e] TCP out of order

[Dennis Ferguson]

> >1) why TCP behaves so badly, when we use multipath network with a small 
> >delay difference in one link ??
> >the throghput drops gradually.
> 
> How large is the delay difference measured in bits?  Is it big enough to
> cause ack reordering (as the message subject implies) or even packet
> reordering?

Note that you can get very consistent packet reordering even if the
speed-of-light delay difference is zero and the circuit bandwidths
are identical, if the packets being generated are sometimes different
sizes.  That is, if you send a big packet followed by a small packet, the
small packet will always catch up to and pass the big packet if they
traverse enough store-and-forward hops.

I'm not positive but I think a smaller packet can be generated consistently
when the sender fills the receive window and the receive window is not
an even multiple of the MSS.  If the short packet consistently arrives
earlier than its longer predecessor(s) then the reordering will be
perfectly correlated with the state of the TCP session and I wouldn't
be surprised if this could cause anomalously poor performance compared
to what one might expect from purely random reordering behaviour.

Dennis Ferguson