[e2e] was double blind, now reproduceable results

[Jonathan Stone]

In message <40ACD7E8.3060004 at isi.edu>Joe Touch writes

[...]

>>>I don't disagree with making all the data available, and that is almost 
>>>always possible, even where reproducibility is hard. 
>> 
>> With the notable exception of whole-payload packet traces.  If you can
>> get permission to collect them at all, its well-nigh impossible to
>> redistribute them. For obvious and excellent reasons.
>
>It ought to be possible in those cases - if not required - to provide 
>reasonable access to run second-party tests to confirm results.
[...]


Oh, absolutely. I've been on both sides of that fence, and that sounds
reasonable to me.  Much of the data I published was from researchers
(such as Vern) who kindly ran my tools ``under escrow'', against their
own traces.  I never saw the raw traces. The tools were validated
against synthetic test data, and against my own traces.

But getting permission to collect whole-packet traces may well impose
conditions more restrictive than you, or I, or Jon might wish from
solely a resarch perspective.

Stanford stipulated I could not disclose *anything* about my
whole-packet traces, without first clearing it with the appropriate
campus authorities.  For example, they got veto rights over early
drafts of my papers, to ensure I didnt disclose anything untoward.
That same veto would apply if I were to run third-party tools over
those traces. Getting permission to run second-party tests might
be a difficult, time-consuming task.

Traces collected from ISPs may (if the ISP is also a telco) also have
to pass the `statistical measurement only' regulatory requirements for
telcos; but I have no idea what those are, these days.